More Agents Is All You Need: LLMs performance scales with the number of agents
Running large language models multiple times and aggregating their answers can significantly improve accuracy on reasoning tasks, according to a new paper that treats each run as an “agent” and selects the most common or similar output. Commenters relate this to classic ensemble methods and beam search, noting that most gains seem to come from a modest number of runs and that further improvements face steep compute, cost, and energy trade‑offs. The findings fuel broader debates about whether simple multi-sampling beats more elaborate agent frameworks, what this implies for paths to AGI, and how swarms of LLM-based agents might eventually affect online discourse and governance.
Method and relation to “agents”
- Paper’s “agents” are just multiple independent LLM runs on the same query (temp > 0), no shared context.
- Outputs are clustered with a similarity metric; the most common/central answer is selected.
- Some commenters argue this largely explains gains seen in more elaborate “multi‑agent” or debate frameworks, which may just be re‑sampling plus selection. Others say the paper treats this as orthogonal and additive to chain‑of‑thought or debate, not a replacement.
Ensembles, MoE, and classical parallels
- Many see this as classic ensemble learning / voting or Monte Carlo sampling rediscovered for LLMs.
- Comparisons to mixture‑of‑experts, random forests, and weather forecasting ensembles are frequent.
- Some note this echoes cognitive theories like “society of mind” and internal sub‑agents.
Performance gains and diminishing returns
- Reported gains: noticeable improvement up to ~10–20 agents, then flattening; one critique notes ~35× more reasoning for ~10% accuracy gain.
- Other work (linked in thread) suggests majority voting can eventually decrease quality, indicating a non‑trivial optimum.
Temperature, randomness, and sampling
- Discussion on why multiple stochastic runs can beat a single low‑temperature run:
- Broader exploration of the model’s probability landscape, escaping local maxima.
- Majority/consensus may approximate the “true” answer while hallucinations are more varied.
- Another cited paper claims temperature itself doesn’t strongly affect problem‑solving, making the mechanism somewhat unclear.
Cost, practicality, and UX
- Major concern: inference costs and latency scale roughly with the number of agents.
- Some argue 10× GPT‑3.5 calls might still be cheaper than one GPT‑4; others doubt 3.5 ensembles can match 4, especially where 3.5 just lacks capability.
- Skepticism that this will see wide adoption in real products due to token waste and slow responses.
Implications, risks, and limitations
- Speculation about swarms of agents online (e.g., karma farms, reputation attacks) and a “dark forest” internet.
- Debate over whether such multi‑agent setups imply progress toward AGI or just expensive simulation tricks.
- Several note that this mostly improves answers in dense knowledge regions and may not help where the base model is systematically wrong.
Meta and terminology
- Complaints about overused “X is All You Need” titles and nonstandard use of “agent” for plain LLM calls.