Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

Original Article ↗ Hacker News Discussion ↗

A hobbyist experimenter reports outperforming major open LLMs on the HuggingFace leaderboard by “brain surgery” on existing models: duplicating specific mid‑stack transformer layers on a pair of gaming GPUs instead of retraining or fine‑tuning. Commenters connect this to ideas like mixture‑of‑experts, looped/recurrent depth transformers, and deep equilibrium models, noting that only certain contiguous blocks of layers improve reasoning—suggesting emergent functional “organs” for translation, reasoning, and output. The work raises broader questions about latent-space reasoning, test‑time compute scaling, and whether future models should be architected around reusable reasoning circuits or pluggable “knowledge banks” rather than monolithic depth.

Relationship to Existing Architectures

  • Several commenters relate the technique to Mixture-of-Experts, looped / recurrent LLMs, and models like Ouro-LLM, LoopLM, and SOLAR that duplicate or reuse layers.
  • OP’s method is described as orthogonal to MoE: it works on dense and MoE models by repeating vertical chunks of the stack, not sparsifying experts.
  • Others note that adding/swapping/duplicating layers has prior art (ResNets, StyleGAN, upcycling), and recent papers argue that pre-LN transformers make middle layers near-identity, concentrating “real computation” in the middle.

Middle Layers as “Organs” / Functional Circuits

  • A key theme is that contiguous mid-layer blocks behave like emergent “organs” or circuits: duplicating whole blocks improves performance, but single layers or arbitrary mixes do not.
  • Heatmaps across layers are interpreted as showing boundaries between such organs (e.g., input encoding, “reasoning”, output decoding).
  • Commenters link this to CKA analyses and other work showing neighboring middle layers have similar representations and residual connections preserve a stable latent space.
  • There is debate whether these patterns are universal structures or artifacts of particular training procedures.

Base64 and Latent “Thought Language”

  • Many are struck by the observation that LLMs can read/write base64 or hex, reason over it, and convert back, despite seemingly limited exposure to such text.
  • Some argue models have likely seen enough base64 in web and email corpora; others stress that the behavior still implies an internal “translation circuit” that maps encoded text into a common reasoning space.
  • This motivates the broader hypothesis of a shared latent “thought language” used across modalities and encodings.

Experiments, Tools, and Limitations

  • Duplicating individual layers or repeating the same block many times generally hurts performance; gains appear only for specific mid-blocks and limited repetition.
  • Multiple disjoint duplicated regions and meta-models (e.g., XGBoost) to predict good merges have been tried, but details are deferred to future posts.
  • Combinatorial explosion is a recurring concern when considering arbitrary reordering or routing between layers.

Speculation and Future Directions

  • Ideas raised include:
    • Looping specific reasoning blocks versus whole-model loops.
    • Dynamic routing that chooses which layer or block to apply next.
    • Variable-depth inference (“how hard to think” knob per token).
    • Pluggable “knowledge banks” or standardized encode/logic/decode modules.
    • Combining organs from different models, or adding new modalities via surgery.
  • Commenters note that hobbyist “LLM brain surgery” is exploring spaces corporate and academic work may have deprioritized due to cost or focus.

Community Reaction and Open Questions

  • The thread is overwhelmingly enthusiastic about the ingenuity, clarity of the writeup, and the sense of “poking a synthetic brain.”
  • Some view the findings as surprising and under-appreciated; others see them as a natural consequence of residual architectures and known optimization behavior.
  • Open questions include how general these organs are across tasks, models, and sizes, and whether training with loops from the start would outperform post-hoc surgery.