LLM4Decompile: Decompiling Binary Code with LLM
Researchers are experimenting with LLMs fine-tuned to turn compiled binaries back into C code, raising the prospect of more powerful, partially automated decompilation tools. Commenters note that while models like GPT‑4 can already produce compilable code from assembly, they struggle with faithfully reproducing program semantics, and current benchmarks (re-compilability vs. re-executability) show significant gaps. The conversation explores potential applications in reverse engineering, deobfuscation, and security analysis, while emphasizing legal, reliability, and evaluation challenges and the likely need to combine LLMs with traditional decompilers and formal verification.
Model performance & training
- Smaller 6B model reportedly outperforms a 33B one; commenters attribute this to undertraining of large models and sensitivity to data quality, LR schedules, etc.
- The decompiler model is fine-tuned from DeepSeek-Coder; from-scratch training is seen as infeasible given only ~4B tokens vs trillions for general LLMs.
- Rough hardware needs discussed: ~2GB VRAM for 1B params, ~14GB for 7B, ~80GB for 33B.
Comparison with GPT‑4 and other LLMs
- GPT‑4 does surprisingly well at decompilation despite no task‑specific training.
- It produces syntactically valid (recompilable) C more often than the specialized model but is worse on semantic correctness (re‑executability).
- People also report using GPT‑4/3.5 for deobfuscating JavaScript and beautifying code, with partial but not complete success.
Relation to traditional decompilers & reverse engineering
- Several note that modern native-code decompilers (IDA, Ghidra, Binja, rev.ng, etc.) are strong baselines and should be part of benchmarks, using prior metrics like gotos, cyclomatic complexity, and graph-edit distance.
- Some practitioners argue decompilation of complex binaries, especially hand‑written assembly or heavily optimized code, likely can’t be fully automatic; tools mainly amplify expert effort.
- Others point out compiler diversity, non‑determinism, and lack of bijection between binaries and source code as hard limits; exact round‑tripping is often impossible.
Evaluation, correctness, and “hallucinations”
- Project currently measures:
- Re‑compilability: does the generated C compile?
- Re‑executability: does the recompiled binary pass test cases?
- Re‑executability rates are low (≈10–15%), even when recompilability is much higher, underscoring semantic gaps.
- Debate over whether round‑tripping or identical binaries is a realistic metric; many stress test quality and fuzzing as more practical checks.
- Some propose formal verification: LLM outputs code plus a proof; a non‑LLM verifier checks equivalence. Others question feasibility and undecidability limits.
Use cases, scope, and future directions
- Suggested uses: aiding reverse engineering, security analysis, malware and obfuscated JS analysis, toolchain detection, lifting to intermediate representations (e.g., RzIL), and author attribution experiments.
- Many see LLMs as “copilots” layered on top of existing disassemblers/IRs rather than standalone decompilers.
- Dataset generation from open-source C (and possibly synthetic code) is seen as a key advantage of this research direction.