Progress on provable inference for large language models

Anthropic’s work on “provable inference” for large language models—using zero-knowledge proofs to cryptographically verify that a specific model and weights produced a given output—is seen as a meaningful step for auditability, routing transparency, and potential regulation of AI compute. Commenters debate how useful this really is for end users who already must trust providers with their data, noting the huge performance overhead, limited current practicality, and the possibility that it mainly protects vendors’ business models or enables regulatory capture. Others highlight emerging use cases such as third‑party evaluators, supply‑chain security for AI services, and future extensions to proving how models were trained, while also raising concerns about attribution and open‑source research being absorbed into proprietary systems.

What Provable Inference Tries to Solve

  • Goal: let a client verify that a specific model (specific weights) produced a response, even over untrusted infrastructure or via third-party routers.
  • Motivations cited:
    • Independent evals needing assurance the evaluated model is the one served.
    • Customers of routing/proxy services wanting proof they got the model they paid for.
    • Detecting token tampering or model substitution (e.g., cheaper, quantized, or “lobotomized” variants).

Debate on Practical Value and Threat Model

  • Supporters see it as key for:
    • AI supply-chain security and “defense in depth”.
    • Auditing and assurance firms doing black-box evaluation.
    • Preventing fraud by resellers or shady routers.
  • Skeptics argue:
    • If you don’t trust the provider with your data, verifying weights is the least of your problems.
    • It doesn’t address privacy, data exfiltration, or correctness of answers.
    • For many users, it’s a niche or internal concern, not a top practical problem.

Technical Limits and Performance

  • Current system: integer-only weights, greedy decoding, ~1k context length.
  • Proof generation is ~70–95× GPU time of normal inference; verification is ~17–21s CPU and ~6.5 MB per commitment.
  • Some see this as progress but still far from production viability for general workloads.

Quantization, Substitution, and Open Weights

  • Discussion on whether it can catch hosts silently using quantized or downgraded models.
  • For open-weight models, some argue recomputing deterministically can be cheaper than ZK proofs.
  • Attestation could in principle prove exact checkpoint and quantization; this system currently mainly supports integerized versions.

Comparison to Alternatives

  • Compared with:
    • Hardware confidential computing / remote attestation (seen by some as more practical but hardware-dependent).
    • Watermarking (proves “came from this provider,” not “from these weights on this input”).
    • Multi-provider recomputation and HTTPS for integrity in simpler threat models.

Regulatory and Business Implications

  • Some view this as enabling regulatory regimes that monitor hardware use (inference vs training) and as potential regulatory capture / lock-in.
  • Others see it as a way to maintain price premiums for “premium” models when routers can otherwise substitute cheaper ones.

Training Provenance and Attribution

  • Several commenters highlight that provable training (proof a model was trained only on certain data) would be far more impactful but is much harder.
  • There is a side discussion about prior academic work (e.g., VerInf), licensing (MIT vs GPL), and perceptions of inadequate attribution in Anthropic’s write-up.