Some thoughts about Anthropic's new cryptanalysis results
Anthropic’s new cryptanalysis results on AES, achieved largely by telling an unreleased model to “keep going,” are prompting people to reassess both the capabilities of modern LLMs and the value of elaborate prompt engineering. Commenters highlight how simple, iterative prompts can now surface nontrivial math and security findings, while also debating whether “glorified autocomplete” remains a fair characterization of models that increasingly act like mechanized attention across the entire corpus of human knowledge. The thread widens into questions about what counts as intelligence or AGI, the limits of current models, and how quickly these tools are reshaping fields like cryptography, cybersecurity, and software development.
Prompting vs. “Process Engineering”
- Several comments note that the cryptanalysis results and recent math breakthroughs came from extremely simple prompts like “keep going” or “you should do a breakthrough.”
- This leads some to question elaborate “prompt engineering” practices, arguing that current models respond well to short, natural instructions.
- Others frame this not as prompt engineering but as “process engineering”: designing iterative workflows and feedback loops, where domain experts can systematically push models to better results.
Model Capability, Limits, and Filters
- Commenters highlight that the AES work underscores how powerful current models are at systematically applying known techniques, not inventing exotic new ones.
- Many push back against the “glorified autocomplete” dismissal, arguing that next-token prediction can yield surprisingly deep reasoning and discovery, especially with post-training and tools.
- Others defend the phrase as a useful ontological reminder that these are statistical text engines, not sentient agents, though outcomes can still be remarkable.
- There is frustration with strong safety filters and automatic downgrades (e.g., from Mythos/Fable to weaker models) on seemingly harmless biology or history questions.
Intelligence, Consciousness, and AGI Definitions
- Long subthread debates what “intelligence,” “consciousness,” and “AGI” actually mean.
- Some argue that being a next-token predictor does not rule out advanced reasoning or even consciousness; others say LLMs clearly lack continuous experience and other key properties.
- AGI is variously defined as: human-level performance across most tasks, self-improving superintelligence, or simply current LLMs already matching median human text competence.
- There is consensus that goalposts have been fuzzy and that LLMs exposed how many incompatible AGI definitions coexist.
LLMs as “Mechanized Attention”
- One analogy: LLMs are like “mechanized attention” or an ultra-fast research librarian, surfacing and recombining vast amounts of prior knowledge.
- This may “raise the floor” of solution quality by making deep prior work accessible even for quick or toy projects.
- Some worry that heavy reliance on LLMs could atrophy human skills and independent thinking.
Cryptography-Specific Concerns
- A few participants note fatigue with “AES is broken” style headlines based on narrow, highly constrained attacks.
- There is concern that AI-generated cryptanalysis will increase low-value, clickbaity crypto posts alongside genuine advances.