AI models can provably hide arbitrary capabilities
Researchers claim that neural networks can be engineered to contain “encrypted” behaviors or backdoors that only activate on secret triggers, even when model weights are fully visible. Commenters explore the security and trust implications for both open and closed models, noting that traditional vetting, benchmarks, or static analysis may be unable to reveal such hidden capabilities. The thread raises concerns about data poisoning, model supply-chain attacks, and the limits of alignment and sandboxing as defenses.
Backdoors, Sleeper Agents, and Triggered Behavior
- Many commenters see the work as formalizing a long-feared scenario: models embedding “sleeper” behavior that only activates on secret triggers.
- Examples discussed include prompts that look normal vs. prompts with a secret token that cause exfiltration-like outputs or tool-use suggestions.
- Some argue this is analogous to classic backdoors or polymorphic/encrypted malware embedded inside weights.
- Others emphasize that models themselves only emit text; actual harm requires insecure harnesses/agents that execute or act on outputs.
Detectability, Theory Limits, and Obfuscation
- Some initially claim such backdoors are “trivially detectable” via atypical weight distributions.
- Others counter with cited work on obfuscating weight distributions and note that the paper itself describes ways to better hide the structures.
- Rice’s theorem and related undecidability results are invoked: any powerful model will have non-trivial properties that can’t be reliably detected by static analysis alone.
- Several people stress that benchmarks alone cannot guarantee absence of hidden capabilities.
Open vs Closed Models, Training Pipelines, and Trust
- Debate over whether open weights or fully open training pipelines provide meaningful protection.
- Some say even full access to data and code doesn’t help unless you can retrain and verify weights; for most, that’s infeasible.
- Others argue open weights at least allow partial scrutiny or weight hashing against a “known good” version.
- General consensus: this is fundamentally a publisher-trust problem; both state actors and corporations could, in principle, embed backdoors.
Robustness, Fine-Tuning, and Hidden Knowledge
- Thread notes claims that such “encrypted” circuits can be very robust to fine-tuning and noise, sometimes more so than normal language capabilities.
- An anecdotal experiment suggests models can “forget” a bizarre Q&A pair and later spontaneously “rediscover” it, hinting at latent or hidden representations.
- Commenters link this to ideas like superposition, lottery-ticket subnetworks, and analogy to dormant genes reactivating.
Security, Alignment, and Practical Impact
- Some view this as mostly a cool demo; others see it as strong evidence that safety/alignment based only on behavior is fragile.
- A recurring view: alignment is less important than enforceable external constraints and sandboxing.
- Speculation arises about data poisoning to induce such backdoors and about nation-state or corporate misuse, though concrete real-world cases remain unclear.