LLM4Decompile: Decompiling Binary Code with LLM

Original Article ↗ Hacker News Discussion ↗

Model performance & training

  • Smaller 6B model reportedly outperforms a 33B one; commenters attribute this to undertraining of large models and sensitivity to data quality, LR schedules, etc.
  • The decompiler model is fine-tuned from DeepSeek-Coder; from-scratch training is seen as infeasible given only ~4B tokens vs trillions for general LLMs.
  • Rough hardware needs discussed: ~2GB VRAM for 1B params, ~14GB for 7B, ~80GB for 33B.

Comparison with GPT‑4 and other LLMs

  • GPT‑4 does surprisingly well at decompilation despite no task‑specific training.
  • It produces syntactically valid (recompilable) C more often than the specialized model but is worse on semantic correctness (re‑executability).
  • People also report using GPT‑4/3.5 for deobfuscating JavaScript and beautifying code, with partial but not complete success.

Relation to traditional decompilers & reverse engineering

  • Several note that modern native-code decompilers (IDA, Ghidra, Binja, rev.ng, etc.) are strong baselines and should be part of benchmarks, using prior metrics like gotos, cyclomatic complexity, and graph-edit distance.
  • Some practitioners argue decompilation of complex binaries, especially hand‑written assembly or heavily optimized code, likely can’t be fully automatic; tools mainly amplify expert effort.
  • Others point out compiler diversity, non‑determinism, and lack of bijection between binaries and source code as hard limits; exact round‑tripping is often impossible.

Evaluation, correctness, and “hallucinations”

  • Project currently measures:
    • Re‑compilability: does the generated C compile?
    • Re‑executability: does the recompiled binary pass test cases?
  • Re‑executability rates are low (≈10–15%), even when recompilability is much higher, underscoring semantic gaps.
  • Debate over whether round‑tripping or identical binaries is a realistic metric; many stress test quality and fuzzing as more practical checks.
  • Some propose formal verification: LLM outputs code plus a proof; a non‑LLM verifier checks equivalence. Others question feasibility and undecidability limits.

Use cases, scope, and future directions

  • Suggested uses: aiding reverse engineering, security analysis, malware and obfuscated JS analysis, toolchain detection, lifting to intermediate representations (e.g., RzIL), and author attribution experiments.
  • Many see LLMs as “copilots” layered on top of existing disassemblers/IRs rather than standalone decompilers.
  • Dataset generation from open-source C (and possibly synthetic code) is seen as a key advantage of this research direction.