AI-powered conversion from Enzyme to React Testing Library

Original Article ↗ Hacker News Discussion ↗

Slack’s use of large language models and AST-based tooling to convert 15,000 React unit tests from the obsolete Enzyme framework to React Testing Library prompts debate over how effective and cost‑efficient such automation really is. Commenters scrutinize Slack’s reported metrics (22% fully auto‑converted tests vs. 80% “accurately converted” content), question whether simpler codemods, manual edits, or even maintaining Enzyme might have been cheaper, and highlight the risk of generating tests that pass without truly testing anything. The exchange reflects broader skepticism about AI‑branded engineering efforts, the fragility of tooling tightly coupled to React internals, and the growing market for hybrid human/automation approaches to large‑scale code migrations.

Project Overview & Approach

  • Slack migrated ~15k React tests from Enzyme to React Testing Library (RTL) using a pipeline that combines AST-based transforms and LLMs.
  • Rule-based codemods were tried first; complexity and edge cases exploded, so LLMs were added as a higher-level transformer on AST-structured input.
  • Manual review of converted tests remains required.

Automation Effectiveness (22% vs 80%)

  • One subset: ~2,300 tests → ~500 fully auto-converted and passing (22%). Slack equates this to ~22% developer-time savings for that slice.
  • Another evaluation: for selected files, about 80% of content was judged “accurately converted,” with ~20% needing manual intervention.
  • Commenters debate whether “80% automatically converted tests” is a fair summary, or whether the more honest headline is that only ~22% of tests were fully auto-migrated.

Codemods, Vim Macros, and “Dumb” Tools

  • Some argue that interactive text tools (e.g., Vim macros) or well-crafted AST codemods could achieve 60–80% coverage for this kind of mechanical migration.
  • Others counter that codemods were explicitly tried and hit complexity limits, and Vim macros are not more powerful than AST-based codemods, just more interactive.

Enzyme vs React Testing Library & Migration Strategy

  • Enzyme is considered effectively abandoned and tightly coupled to React internals; it doesn’t support newer React versions without major effort.
  • Some suggest it might have been cheaper to fund Enzyme/react-18 support or adopt community adapters.
  • Others argue maintaining such a deep-internals test framework is riskier and more expensive long term than moving to RTL’s more “user-centric” API.

Test Quality & “Are These Tests Still Testing Anything?”

  • Multiple commenters question using “test passes” as the main success metric; a migrated test might pass while no longer asserting the same behavior.
  • Mutation testing is raised as a way to evaluate whether the test suite still catches real defects.
  • Some share experience that RTL can make it easy to write tests that always pass unless carefully designed.

AST vs CST Terminology

  • One thread questions whether “AST” is used correctly, suggesting a CST-like structure is needed to preserve formatting.
  • Responses note that real-world syntax trees blur AST/CST boundaries; tools often keep only the syntax details they care about and rely on formatters to normalize whitespace.

Media & Hype Skepticism

  • Several comments criticize the secondary InfoQ article as spinning or misreading Slack’s numbers to hype AI.
  • Others point out that the InfoQ framing isn’t obviously wrong given Slack’s own ambiguous statistics and marketing tone.
  • Broader frustration is expressed with AI hype cycles and management pressure to showcase AI use, even when it’s only part of a larger, fairly standard migration effort.