More Agents Is All You Need: LLMs performance scales with the number of agents

Method and relation to “agents”

• Paper’s “agents” are just multiple independent LLM runs on the same query (temp > 0), no shared context.
• Outputs are clustered with a similarity metric; the most common/central answer is selected.
• Some commenters argue this largely explains gains seen in more elaborate “multi‑agent” or debate frameworks, which may just be re‑sampling plus selection. Others say the paper treats this as orthogonal and additive to chain‑of‑thought or debate, not a replacement.

Ensembles, MoE, and classical parallels

• Many see this as classic ensemble learning / voting or Monte Carlo sampling rediscovered for LLMs.
• Comparisons to mixture‑of‑experts, random forests, and weather forecasting ensembles are frequent.
• Some note this echoes cognitive theories like “society of mind” and internal sub‑agents.

Performance gains and diminishing returns

• Reported gains: noticeable improvement up to ~10–20 agents, then flattening; one critique notes ~35× more reasoning for ~10% accuracy gain.
• Other work (linked in thread) suggests majority voting can eventually decrease quality, indicating a non‑trivial optimum.

Temperature, randomness, and sampling

• Discussion on why multiple stochastic runs can beat a single low‑temperature run:
  • Broader exploration of the model’s probability landscape, escaping local maxima.
  • Majority/consensus may approximate the “true” answer while hallucinations are more varied.
• Another cited paper claims temperature itself doesn’t strongly affect problem‑solving, making the mechanism somewhat unclear.

Cost, practicality, and UX

• Major concern: inference costs and latency scale roughly with the number of agents.
• Some argue 10× GPT‑3.5 calls might still be cheaper than one GPT‑4; others doubt 3.5 ensembles can match 4, especially where 3.5 just lacks capability.
• Skepticism that this will see wide adoption in real products due to token waste and slow responses.

Implications, risks, and limitations

• Speculation about swarms of agents online (e.g., karma farms, reputation attacks) and a “dark forest” internet.
• Debate over whether such multi‑agent setups imply progress toward AGI or just expensive simulation tricks.
• Several note that this mostly improves answers in dense knowledge regions and may not help where the base model is systematically wrong.

Meta and terminology

• Complaints about overused “X is All You Need” titles and nonstandard use of “agent” for plain LLM calls.