Stealing Reasoning Traces from Proprietary LLM APIs

Original Article ↗ Hacker News Discussion ↗

Researchers have shown that encrypted “chain-of-thought” reasoning traces from proprietary LLMs like OpenAI, Anthropic, and Google can be replayed into weaker sibling models, which can then be jailbroken to reveal the stronger model’s hidden reasoning in plaintext. Commenters examine how this works technically, why providers used portable encrypted blobs in the first place, and what mitigation options (per-model keys, disabling model switching, server-side storage) would mean for usability and zero-data-retention promises. The thread also dives into the ethics and legality of using these recovered traces for distillation, questioning whether accessing reasoning you paid token costs for can reasonably be called “stealing.”

Attack mechanism & findings

  • Encrypted chain-of-thought (CoT) blobs can be replayed into weaker “sibling” models that share the same decryption layer.
  • The provider decrypts the blob server-side and feeds raw CoT into the model’s context; a jailbroken weaker model can then be prompted to echo it back in plain text.
  • This does not break the cryptography; it abuses the product design (cross-model, cross-session replay).
  • Commenters note similar CoT “grug-speak” patterns and confirm that decrypted traces match what vendors have previously shown.

Design choices: statelessness & model switching

  • Encrypted CoT exists so APIs can be “stateless” and still preserve reasoning across turns and model switches, and to support zero-data-retention modes.
  • Cross-model reuse is considered intentional; otherwise users couldn’t downgrade/upgrade models mid-conversation.
  • Some argue the true flaw is not binding traces tightly to a user/session.

Potential fixes and limitations

  • Proposed mitigations: per-model keys, per-user keys, refusing downgrades, stripping traces on model change, or adding access-control metadata.
  • Others counter that encryption details are irrelevant: any model that sees raw CoT can be tricked into revealing it.
  • Providers have reportedly patched the specific attack, but details are unspecified and some fear future “fixes” will hurt usability (e.g., locking model choice).

Distillation, competition, and Kimi K3

  • Many see this as a powerful distillation channel: use expensive frontier models for normal work, then harvest CoT via cheaper models.
  • Evidence discussed that a prominent non-Western model aligns unusually well with certain proprietary CoT, interpreted by some as strong evidence of prior distillation; others argue that Chinese labs also made major independent advances and that “distillation explains everything” is overblown.

Ownership, “stealing”, and ethics

  • Heated debate over whether accessing CoT is “stealing” given that:
    • Users pay per token, including hidden reasoning tokens.
    • LLM outputs are often not copyrightable, especially in some jurisdictions.
  • One side: vendors invested heavily to produce proprietary reasoning traces and clearly signal intent to protect them (encryption, ToS).
  • Other side: models are trained on the world’s un/under-compensated IP; calling CoT use “theft” is seen as hypocritical and as IP-expansion rhetoric.
  • Distinction raised between legal theft vs. contract breach vs. espionage vs. simple unauthorized copying.

User experience, transparency, and safety tradeoffs

  • Many want direct access to reasoning for debugging, trust, prompt tuning, and to avoid paying for invisible tokens.
  • Others are increasingly willing to accept opaque reasoning as a pragmatic tradeoff: vendors can manage complex reasoning agents better, and users can focus on applications.
  • There is frustration that security/safety responses often reduce useful features (e.g., removal of full logprobs, potential future bans on model switching).

Security, privacy, and data retention

  • Some argue this isn’t a classic “safety” failure but an opacity/IP-protection failure; others tie it to broader safety concerns (e.g., avoiding unsafeguarded distilled models).
  • Concern that encrypted blobs may contain PII, secrets, and credentials, and that these can now be surfaced.
  • Zero-data-retention contracts constrain server-side fixes; client-side blobs were partly meant to avoid storing such data at all.
  • Commenters note that API providers likely already log full CoT in many non-ZDR settings, so the “encryption” is more about user visibility than true secrecy.

Tool-use variants & other technical notes

  • A related trick: disable built-in reasoning, provide a custom “deep_think” tool, and ask the model to place its internal reasoning into tool arguments, effectively exposing CoT.
  • Some see this as functionally similar to the replay attack, others distinguish it as more of a tool-spec abuse than a cross-model encryption flaw.

Benchmarks, training data, and traces

  • Observations that models sometimes appear to “know” benchmark solutions in advance; CoT and API summaries can blur the line between genuine reasoning and memorized derivations.
  • Some note that once benchmarks are public, it’s almost guaranteed they end up in training data; the real question is whether training was explicitly optimized for them.

Meta: site and HN UX

  • Many praise the paper’s aesthetics but criticize the website as hard to read, especially on mobile (rotated text, overlapping images, awkward scrolling).
  • Side discussion about HN’s submission UX: initial “description” text for link posts is actually posted as an undistinguished top-level comment, causing confusion for submitters and readers.