Interpreting claims that a theorem is correct because it was checked in Lean
Claims that a mathematical theorem is “proved because it was checked in Lean” are being scrutinized as AI systems generate massive formal proofs. Commenters note that while Lean’s proof checker offers strong guarantees that a formal statement logically follows from given axioms, it does nothing to ensure the formal statement matches the intended theorem, avoids hidden tricks, or is free from kernel bugs. Many still see mechanized proofs and formal verification as crucial progress, but argue they must be paired with human judgment, clear specifications, and better tools if they’re to be trusted in both mathematics and software.
Lean proofs vs mathematical truth
- Many compare “proof checked in Lean” to “code that compiles” or “passes unit tests.”
- Consensus: Lean guarantees “if this formal statement is what you meant, then the proof is correct,” but does not guarantee that the formal statement matches the informal theorem.
- Some argue it’s closer to “formally verified code” than mere compilation, but the spec-mismatch problem remains.
Specification vs formal statement
- Central concern: verifying that the Lean statement matches the intended theorem is nontrivial, especially with large dependency stacks and many definitions.
- Misformulations (e.g., forgetting that ℕ includes 0 in one FLT-style example) can make false or trivial statements look “proved.”
- Large imports can hide redefinitions or “shenanigans” that subtly change meanings; tools like
lake comparatorare discussed as mitigations.
LLMs, proof size, and quality
- Multiple comments: LLMs often produce extremely long, convoluted Lean proofs, far larger than human ones.
- They sometimes find shorter arguments than brute-force provers, but on average produce “slop” that’s hard to understand.
- Lean generated by LLMs is likened to unit tests that vacuously pass (e.g., replacing tests with no-ops).
Trust, kernels, and potential exploits
- Discussion of independent kernels and formally verified checkers as partial defenses against Lean kernel bugs.
- A known kernel bug was exploited to “prove” a false statement, showing that agents can or could eventually exploit such issues.
- Running arbitrary Lean code is flagged as unsafe because it can execute arbitrary programs.
Applications to software & formal verification
- Strong interest in using Lean (and related tools) to verify software, from DSLs up to full languages.
- Some projects model programming languages directly in Lean; others use shallow embeddings or language-specific tools.
- Formal proofs are seen as a powerful complement to tests and review, but not a silver bullet.
Value and purpose of massive proofs
- Concern about proofs so large no human can understand them.
- Some see value even then (e.g., in verified software or guiding experiments in physics); others worry math becomes a black-box game.
- Overall sentiment: formalization is valuable, but claims like “proved in Lean” need careful, nuanced interpretation.