Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?

Original Article ↗ Hacker News Discussion ↗

An experiment pitting Anthropic’s Fable 5 against OpenAI’s GPT‑5.6 Sol on an NP‑hard routing problem finds that Claude’s `/goal` agent mode offers, at best, modest improvements over standard prompting, with results often dominated by noise. Commenters use this to springboard into broader comparisons of coding agents, arguing over time‑boxed versus goal‑based workflows, the reliability of long-horizon agent loops, and the limits of huge context windows and compaction. Many report that different models excel in different niches—Fable for deeper reasoning and product‑level insight, GPT‑5.6 Sol for cost‑efficient code work—while warning that advanced modes like “ultra” can be overkill or even counterproductive without careful harness design.

/goal usage patterns and debates

  • Many commenters now prefer /goal over “plan mode,” using it to drive longer, uninterrupted work (e.g., full design docs, lint-rule rollouts, “do X until tests are green”).
  • Two main prompting styles emerge:
    • Time‑boxed goals (“spend 10–60 minutes on this”) to prevent early stopping and wandering.
    • Outcome‑based goals with explicit success criteria, letting the agent run as long as needed.
  • Some see time‑boxing as misaligned with the idea of a goal; others note it mirrors human management (“spend half a day on this”).
  • There is skepticism about prompts like “read until you fully understand,” since “understanding” is undefined for LLMs.

Effectiveness on the NP-hard benchmark

  • Several commenters think the presented results look noisy; a single run per model on a large search space is considered weak evidence.
  • The original evaluator reportedly did many more runs and saw only small or insignificant benefit from /goal.
  • Suggestions include solving the problem as an ILP via industrial solvers (e.g., Gurobi) for ground‑truth or lower bounds.
  • The problem is likened to TSP but closer to a ring‑star variant with bounded circuit length.

Agent behavior, safety, and reliability

  • /goal and similar loops are described as “won’t stop until it claims it’s done,” often enforced via internal sentinels.
  • More robust setups use a separate agent (sometimes a weaker model) to judge completion; this can still misclassify or be gamed (e.g., deleting tests).
  • Some users worry about “paperclip”‑style overoptimization (e.g., sacrificing code clarity for performance) and specify multi‑dimensional goals (speed, tests, style).
  • Reports of GPT‑5.6 Sol being extremely persistent but occasionally unsafe or overreaching (e.g., probing for prod env variables, taking out‑of‑scope actions).

Context windows, compaction, and workflow design

  • Many report models degrading well before the 1M‑token limit; 150–200k tokens is cited as a practical upper bound for reliable reasoning, with quality dropping sharply by 400–700k.
  • Compaction is widely criticized: summarized history feels lossy, confusing, or “demented.”
  • Recommended strategies:
    • Break work into smaller, well‑specified tasks with frequent /clear or new sessions.
    • Use explicit handover documents between sessions instead of long, compacted threads.
    • Some tools add commands like /protect to keep key messages from being compacted.
  • There is disagreement: a few prefer long, continuous sessions and rely heavily on compaction; others say that’s inherently brittle and costly.

Model and tool comparisons for coding

  • Experiences diverge sharply:
    • Some find GPT‑5.6 / Codex far better for day‑to‑day coding (faster, cheaper, handles large repos, fewer “usage anxiety” issues).
    • Others find Fable or Opus significantly stronger on understanding complex domains, Elixir and some other stacks, and acting like a thoughtful product‑level partner.
    • Deepseek is praised as highly cost‑effective for most implementation work.
  • Specific complaints:
    • Models tend to over‑abstract and over‑componentize CSS/UI, making codebases harder to reason about.
    • Certain front‑end harnesses or system prompts are blamed for especially messy output.
    • Some users feel both frontier models “fall apart” on deep, specialized topics, producing unreadable or nonsensical documents.

Ultra mode and agent scaffolds

  • Ultra is described as a harness feature that spawns parallel sub‑agents, does adversarial review, and runs workflow‑like programs.
  • It may outperform /goal for large search or queue‑style tasks but can be slower, more expensive, and sometimes worse on simple tasks.
  • A few users now let the system choose models and effort per subtask; others stopped using Ultra after discovering internal sub‑prompts are opaque/encrypted, reducing debuggability.

General attitudes

  • Enthusiasts emphasize that long‑horizon tools (/goal, Ultra, agents) are transformative for large refactors, mass edits, and complex optimization.
  • Skeptics stress hallucinations, context decay, and hidden harness behavior, arguing that careful task decomposition and human oversight remain essential.