Meta's new LLM-based test generator

Original Article ↗ Hacker News Discussion ↗

AI tools that generate unit tests from existing code are emerging inside big tech, with Meta’s new LLM-based “TestGen” cited as a prominent example. Commenters report that large language models can meaningfully speed up boilerplate work and test generation—especially for routine code or characterization tests on legacy systems—but argue that good tests encode human intent and domain knowledge in ways current models can’t reliably infer. Many see value in using LLMs as “junior developers” or fuzzing-style assistants under strict human review, while warning that over-optimizing for coverage metrics or blindly accepting AI output risks bloated, brittle test suites and harder-to-maintain codebases.

Perceived impact of AI coding assistants

  • Some developers report sizable productivity gains (10–100%+) from copilots for boilerplate, code completion, syntax lookups, and mundane glue code.
  • Others find them marginal or net-slowing, especially for niche domains, complex systems, or when they already type little code relative to design/debug time.
  • Integration in the IDE is generally seen as more useful than chat-style interfaces.
  • AI can help compensate for specific cognitive weaknesses (planning, memory, fatigue), even if time savings are small.

Using LLMs for test generation

  • Many find LLMs very effective at generating unit tests: given code + an example test, they can produce reasonable suites, including edge cases and mocks.
  • Some treat LLMs as “junior devs” proposing tests/PRs that must pass existing checks and then be human-reviewed.
  • LLM-generated tests are likened to fuzzing or characterization tests that expand coverage, especially for legacy or poorly understood code.

Quality, coverage, and value of tests

  • Several argue that tests should encode intent, act as executable specs, and tell a “story”; they doubt LLMs can infer true intent from code alone.
  • Others see clear value in LLMs covering “long tail” and routine error paths developers often skip.
  • There is strong criticism of coverage-chasing: too many shallow tests can calcify code, act as mere change-detectors, and add maintenance burden.

Concerns about feedback loops and metrics

  • Worry that management will push for high coverage metrics, leading to bloated, low-value AI-generated test suites that future devs must appease.
  • Fear of a “doom loop” where AI generates code and tests trained on each other, degrading future training data and obscuring real correctness.

Trust, IP, and security worries

  • Some are wary of sending proprietary code to third-party LLMs; others think the risk is overstated.
  • Concerns are raised about potential supply-chain/backdoor vectors via centralized AI providers.

Interpretation of Meta’s results

  • Commenters note the paper’s stats can be misread: success is reported per test class, not per test case, which likely overstates effectiveness.
  • A single test covering 1,326 lines is seen by some as a fluke; using it to claim “huge value” is viewed as speculative and potentially misleading.