Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

Original Article ↗ Hacker News Discussion ↗

A new multi‑agent “team” setup built on five Qwen3.8‑27B open models reportedly matches or beats Anthropic’s Fable 5 on the LiveCodeBench Hard coding benchmark, sparking interest in whether clever harnesses can let smaller, local models rival proprietary frontier systems. Commenters probe how much of the gain comes from role‑based agents and better context management versus raw model quality, and question the reliability of saturated, older benchmarks like LiveCodeBench. The thread also broadens into concerns about keeping powerful, uncensored open‑weight models viable on consumer hardware amid tightening platform moderation and rapidly advancing closed models.

Benchmark and reported result

  • Memdoor reimplements a previously discussed GVS5H harness using a “team” of five Qwen3.8‑27B agents (planner, ideators, solver, manager, repair).
  • On the first 10 problems of LiveCodeBench Hard, it reports 9.8/10 vs the original harness at 9.6 and Fable 5 at 8.0.
  • Some see this as evidence that open 27B models, with good harnessing, can match or exceed a “frontier” model for coding on this benchmark.
  • Others argue the sample is tiny and LiveCodeBench Hard is old and likely saturated/contaminated, so claims of parity are weak; they want newer, broader benchmarks and speed/cost comparisons.

Harnesses and multi‑agent “teams”

  • Several comments stress that multi‑agent setups and harness engineering (roles, iteration loops, file‑based communication, context management) can significantly boost performance.
  • One view: teams work because they divide labor, constrain context per role, and filter out noisy intermediate tokens, effectively expanding and focusing usable context.
  • Another view: it’s easy to tune a harness to win a specific eval but hard to build one that generalizes; “swarms of idiots” often just loop in parallel.

Model quality, quantization, and alternatives

  • Debate over Bonsai 2‑bit and other very low‑bit quants: some say they’re fast and “not that much dumber,” others say 2–3‑bit quants are clearly worse for real coding, with errors compounding over long dialogues.
  • Higher‑bit (Q5/Q6) or good 3‑bit ISTA quants are seen as usable compromises; claims of “95% of full accuracy” are criticized as benchmark‑cherry‑picking.
  • Many suggest trying Qwen3.8‑Flash‑Next or future Qwen4 variants, which benchmarks suggest are stronger and faster.

Local vs proprietary frontier models

  • Strong desire to keep local, open‑weight models competitive to avoid dependence on hyperscalers and their censorship/safety policies (e.g., blocking game reverse‑engineering, scraping, some web automation).
  • Some argue current open models (Qwen, DeepSeek, GLM) are already “good enough” for most coding if the human is skilled and uses tools/tests.
  • Others counter that proprietary models still solve hard tasks much faster and with fewer iterations, and that open weights will age as their baked‑in knowledge goes stale without ongoing training.

Hardware and feasibility

  • Examples range from consumer laptops (M‑series Macs, gaming GPUs with 16–32GB VRAM) running 27B/Flash‑Next at modest speeds, up to multi‑GPU rigs and specialized chips (Cerebras) for very high throughput.
  • There’s optimism that improving inference engines and quantization will make Qwen‑class models fast and cheap on consumer hardware, but 16GB VRAM remains tight for serious multi‑agent work.

Openness, censorship, and hosting

  • Concern that platforms like Hugging Face may increasingly remove “uncensored” or “abliterated” models; links given to removed cyber‑tuned models and mirrors tracking takedowns.
  • Others argue such models will just migrate to alternative hosts or torrents; the process of “de‑censoring” open weights is already well understood and relatively affordable.

Real‑world use and limits

  • Some report using local Qwen/DeepSeek/GLM for real coding: bug‑finding, small tools, reverse‑engineering and game clones, with good but not frontier‑level autonomy.
  • Skeptics note a lot of current content is “benchmark and tokens per second talk,” and they haven’t yet seen many substantial production systems built primarily on local models.