Brood War Bench

Original Article ↗ Hacker News Discussion ↗

A new “Brood War Bench” uses StarCraft: Brood War as a real‑time benchmark for large language model agents, testing their ability to manage strategy, tactics, and rapid decision-making under tight time and action constraints. Commenters contrast this generalist, tool-using approach with earlier specialized game AIs like AlphaStar and Dota bots, arguing about fairness (APM limits, map vision) and whether current LLMs can ever match human or RL-trained performance in such mechanical games. The thread also branches into related ideas such as extending the framework to other games, combining fast and slow models, and using game AIs as a way to measure the speed–intelligence trade-offs of different models.

Benchmark concept & goals

  • Brood War is seen as a strong benchmark because it stresses long‑term strategy, tactics, orchestration, and real‑time decision-making, unlike many existing benchmarks that mostly test static problem-solving.
  • The benchmark highlights tradeoffs between model strength and latency; some suggest using it to map a “real-time vs batch” performance spectrum and compare models along that frontier.

Agent harness & observed behavior

  • Agents interacted via BW-API style commands and observations; several models treated the RTS as if it were turn‑based, thinking too long and getting overrun.
  • Faster, “weaker” models sometimes outperformed stronger but slower ones, suggesting that speed and action frequency matter as much as raw reasoning.
  • Commenters want more detail on state representation and tools, plus multi‑game series where models can analyze their own replays and pass written “learnings” to fresh agents.

Strategy, races, and game mechanics

  • At high play levels Zerg is considered strongest due to production scaling (multiple units per hatchery), flexible tech switches, and powerful units like mutalisks and especially defilers.
  • Gimmick tactics like Protoss Dark Archon mind control into dual‑race play are viewed as non‑viable in serious competition.

Comparisons to other AI benchmarks

  • People reference SC2’s AlphaStar, OpenAI’s Dota 2 bots, GoBench (9×9 Go vs KataGo), and current Brood War ladder bots.
  • There is debate over fairness: prior systems often used API advantages (map‑wide vision, superhuman APM, perfect selection), or even map hacks, which some see as undermining their value as human‑level benchmarks.

Limitations, fairness, and real-time constraints

  • StarCraft’s high APM and multi‑screen micro make it hard for LLM-based systems, which have significant latency and tend to “overthink.”
  • Some argue RTS games are poor benchmarks because human vs bot comparisons are muddied by mechanical and information asymmetries; others see that as intrinsic to the challenge.

Future ideas & extensions

  • Suggestions include: letting models write their own micro scripts, teaming fast and slow agents, adding click/APM throttles, including more model families (e.g., Gemini, Chinese models), and adapting the idea to other games like Age of Empires II or speed‑limited turn-based games.

Community nostalgia & culture

  • Many share strong nostalgia for Brood War: LAN cafés, clan play, map-making, and its deep cultural impact in places like South Korea.
  • The thread mixes technical discussion with personal anecdotes, underscoring how formative the game was for many players and developers.