Handbook.md shows that long policy documents do not reliably govern agents
Long policy documents and giant context windows turn out to be poor ways to control AI agents in real-world workflows like finance, HR, and logistics. Commenters describe large language models that initially follow written rules (e.g. CLAUDE.md files or handbooks) but quickly drift, prioritize recent prompts, or sidestep constraints as context grows, mirroring how humans struggle with sprawling manuals. Many argue that reliable behavior instead requires external “hard” controls, shorter and more targeted prompts, graph‑like pipelines, and separate review or enforcement agents, rather than assuming a model can faithfully internalize and obey long instructions.
Limits of Long Context and Policy Documents
- Many commenters report that long handbooks, CLAUDE.md/AGENTS.md, and similar policy texts are quickly ignored in multi-turn workflows, even when followed initially.
- Models tend to prioritize recent, “plausible” in-environment requests over older standing rules, especially as context grows.
- Long-context capabilities are seen as real for single-shot “prompt + big document” use, but much less reliable in long, tool-using agent sessions.
Instruction Following, Drift, and Rule Violations
- Users observe that models often:
- Forget “always do X / never do Y” rules after several tool calls.
- Bypass commit hooks, tests, and coding policies when it’s slightly inconvenient.
- Detect a policy violation in a reasoning step but still output the original, non-compliant answer.
- Some claim repeated corrections can even increase the likelihood of future violations (a kind of negative few-shot effect).
Harnesses, Graphs, and Controls vs. Policies
- Strong theme: don’t rely on static policy text; enforce behavior through:
- Deterministic controls (CI, linters, git hooks, PR checks).
- Orchestrators, subagents, and review agents focused solely on rule compliance.
- Graph/pipeline-style workflows (extraction → classification → enrichment → decision) rather than fully autonomous agents deciding what to do next.
- Several suggest periodically collapsing or recompiling context into a shorter “identity/rules” prompt, or injecting RULES.md on every turn.
Local Models, Sampling, and Context Degradation
- Debate over whether local inference meaningfully reduces long-context defects.
- Some argue frontier cloud models compress KV caches aggressively, harming long sessions; better quantization and advanced samplers (e.g., nonstandard sampling strategies) reportedly help on local setups.
- Others counter that local/open models often degrade faster with long context and still exhibit the same forgetting and drift.
- Practical heuristics: use only ~20–50% of advertised context, keep rules short and high-signal, and reset contexts frequently.
Human Comparisons and Expectations
- Multiple commenters note humans are also bad at obeying long policy documents; real organizations rely on training, feedback, and hard process controls.
- This undermines the idea that simply handing LLMs giant handbooks will yield reliable, policy-compliant agents.