Vulnerability research is cooked
Advances in large language models are raising the prospect that automated agents could rapidly find and even weaponize software vulnerabilities across large codebases, potentially reshaping both offensive hacking and defensive security work. Commenters debate whether this leads to an “exploit explosion” that overwhelms open source maintainers already buried in low-quality AI-generated reports, or ultimately favors defenders who can run the same tools in CI to patch issues before release. Many remain skeptical of near-term hype, noting limits in current models, economic and legal constraints, and the enduring difficulty of remediation and secure design, especially for legacy and closed-source systems.
Quality of LLM-Generated Vulnerability Reports
- Older “curl-style” spam reports were mostly low-quality outputs from weaker models with no verification.
- Newer frontier models are described as “scarily good” at finding real, exploitable issues, including complex chains.
- Some participants stress that human spammers used naive prompts and skipped validation, whereas research setups iterate systematically over codebases and add verification stages.
Pipelines, False Positives, and Spam
- Effective pipelines use multi-stage systems: initial LLM scan → secondary LLM or tool-based exploit validation → human sanity check.
- Suggestions include requiring proof-of-concept exploits and automated sandbox testing to filter out slop.
- Consensus that spam reports will continue; maintainers may now face both slop and a rising stream of real issues, with AI also helping triage.
Defenders vs Attackers
- One view: lower exploit-finding cost favors defenders, who can integrate agents into CI (“find vulnerabilities in this PR”) and break exploit chains by fixing any link.
- Counterview: attackers specialize in exploitation, have stronger incentives, and may get a larger effective boost than generalist developers.
- Some argue the net effect still benefits defense if the same models validate patches and scan for regressions; others note many systems won’t be regularly patched or can’t auto-update.
Remediation, Code Quality, and Agents
- Multiple comments: discovery is not the bottleneck—remediation capacity, risk of regressions, and organizational priorities are.
- Debate over “agent loops” that auto-fix bug queues: supporters claim massive productivity; skeptics warn about non-convergence, new bugs, and design decay.
- Distinction emphasized between ordinary bugs and high-severity, reliably exploitable vulns.
Static Analysis, Formal Methods, and Memory Safety
- LLMs are seen as pushing practice closer to a vision of exhaustive static/dynamic analysis and test generation.
- Some think this strengthens the case for memory-safe languages; others note heavily-tested unsafe languages can still work but become less viable as exploit-finding becomes cheaper.
- Formal methods are discussed as powerful but costly and limited in practice; LLMs might help generate tests, contracts, and proofs but won’t eliminate all bugs.
Hype vs Reality
- Enthusiasts cite recent demos where models found nontrivial vulns and even generated working exploits, not just crashers.
- Skeptics find current case studies underwhelming, seeing mostly pattern matching for known bug classes rather than “tectonic shifts.”
- Unclear how far models will go beyond automating existing scanning/fuzzing workflows, and whether this is a step change or another incremental tool.