Show HN: Gemini 4 Argon plays StarCraft
Large language models are being benchmarked by having them write StarCraft: Brood War bots under a tight one-hour time limit, then pitting those bots against established human-written and RL-trained opponents. Commenters examine how well these agents adapt strategies over many games, where they fall short of top bots that rely on superior micro or long-term learning, and whether performance mainly reflects code recall from training data or genuine problem-solving. The benchmark is framed as a broader test of “agentic” capabilities—planning, experimentation, and logistics—while raising questions about harness design, evaluation formats, and how closely such results track other game and coding benchmarks.
Benchmark concept & setup
- LLMs get ~1 hour to write StarCraft: Brood War Protoss bots via BWAPI; those bots are then evaluated in a multi-tier ladder.
- Time pressure is seen as a useful constraint, but some find it “incoherent” because it mixes coding speed, game understanding, strategy, and experimentation into one opaque metric.
- Defenders argue this “end-to-end” style is a feature, analogous to measuring company revenue or fuel efficiency.
Performance vs traditional bots
- LLM-written bots perform surprisingly well, with one model clearing strong A/S-tier Protoss bots and achieving >50% win rates in some tiers.
- Against top self-play RL bot Pluto and very strong scripted bots, win rates drop to single digits; Pluto in particular is described as far out of reach.
- Human-written bots still beat LLM bots overall and top human players still beat those human-written bots.
Strategy, learning, and limitations
- During the 1‑hour window, agents can play practice matches against the evaluation bots (no source access).
- One detailed account describes an LLM bot progressing through:
- Solid but generic macro (1-base Dragoon/Observer, late expansions).
- High-variance all-ins (proxy 2-gate Zealot).
- Exploit-based strategies (Dark Templar into two-base timing with Psionic Storm) tailored to specific opponent weaknesses.
- The LLM mostly simplifies game states and minimizes decision points rather than mastering full-game complexity.
- Against Pluto, poor build orders and lack of macro refinement remain unsolved even after hundreds of games.
Training data, memorization, and “general g”
- Several commenters worry results might just reflect memorizing open-source bots from training data.
- Others note top bots are very large codebases, and shared examples of LLM outputs reportedly do not resemble existing bots.
- Multiple independent game/coding benchmarks (StarCraft, Go, other programming tasks) show the same model ranking order, which some interpret as evidence of a general capability factor, though others note “jagged” weaknesses and poor judgment remain.
APM, human vs AI, and direct play
- Mechanical skill is not the only bottleneck; good decisions matter more than raw APM.
- Specialized Brood War ladder bots with superhuman micro already dominate humans when allowed thousands of actions per minute.
- Running frontier LLMs as real-time players via vision/mouse is considered currently too slow; code-writing benchmarks are seen as more feasible.
Design choices & future directions
- Protoss vs Protoss was chosen to reduce complexity and because Protoss bot tooling is mature.
- Some want similar benchmarks for StarCraft 2, other RTS/FPS titles, or neural-network-based training pipelines where LLMs design and train smaller game-specific models.