Tracking down the 16-year-old WAL-reset SQLite bug

Original Article ↗ Hacker News Discussion ↗

A rare SQLite write-ahead log (WAL) corruption bug, dormant for 16 years, was exposed by Tailscale’s high-concurrency, aggressively checkpointed control-plane workload, prompting them to fund SQLite’s maintainers to track it down and build new VFS-level debugging tools. Commenters examine what this incident implies about SQLite’s reliability under concurrent access, trade-offs versus client–server databases like Postgres, and the limits of even extremely comprehensive testing. The thread also highlights Tailscale’s engineering culture—paying for upstream support, tolerating nonstandard but carefully reasoned architectures, and supporting open-source alternatives such as headscale.

Tailscale’s handling of the bug & open‑source support

  • Many commenters praise Tailscale for buying SQLite professional support and funding the VFS shim that helped isolate the race.
  • Seen as a rare example of long‑term thinking: pay to solve the immediate issue and also to improve tooling for everyone.
  • Some note this model is common in databases (e.g., Percona/EnterpriseDB) and aligns with SQLite’s published pro‑support/consortium offerings.

Identity and authentication choices

  • Tailscale’s SSO‑only design (no username/password) prompts debate.
  • Arguments in favor: avoiding identity‑provider responsibilities, reduced attack surface (credential stuffing, account farms), and aligning individual account security with enterprise customers.
  • Critiques: magic links and SSO can feel clunky; being tied to GitHub/Apple “forever” feels odd, though support can move accounts or add passkey‑only identities.

Headscale, telemetry, and alternatives

  • Headscale (self‑hosted control plane) is cited as a trust‑boosting factor; some run it happily on NixOS.
  • Complaints include needing extra config to disable telemetry and limited opt‑out on iOS (unclear; a Tailscale dev suggests this may have changed).
  • Important features like App Connectors don’t yet work with Headscale, pushing some users to fully open‑source alternatives like NetBird.

SQLite vs. Postgres and architecture choices

  • Debate over whether a large control plane “should” use SQLite versus Postgres/MariaDB.
  • Defenders note SQLite’s design explicitly supports one‑writer/many‑reader patterns and has excellent durability, performance, and testing.
  • Critics argue concurrency is hard, bugs are deep, and multi‑writer systems with built‑in online backups might simplify operations.
  • Tailscale staff in the thread say they chose SQLite early, scaled vertically, and now depend on local‑storage latency; switching would be non‑trivial.

Details and implications of the WAL‑reset bug

  • Bug only affects WAL mode with multiple connections in different threads/processes and fast, manual checkpointing.
  • Tailscale’s aggressive, non‑standard checkpoint strategy for backups made them more likely to hit it.
  • Some note the SQLite changelog and bug text feel understated given real production outages.
  • There’s discussion clarifying the apparent contradiction between “copying more pages than exist” and “pages not written”: the internal counter is wrong, leading SQLite to skip real writes.

Single points of failure and shard design

  • The shard database is a per‑shard SPOF: corruption takes out that shard’s control plane but not the global data plane, since traffic is peer‑to‑peer once established.
  • Some argue eliminating every SPOF via complex distributed consensus could reduce overall reliability; cost/complexity trade‑offs matter.

Testing, formal methods, and AI tools

  • The thread highlights SQLite’s enormous test suite (tens of millions of lines) yet a 16‑year bug still slipped through.
  • Discussion revisits “tests can’t prove absence of bugs,” limits of exhaustive testing, and state‑space explosion.
  • Links are shared to a TLA+ model that can reproduce the issue and to Antithesis’s deterministic concurrency testing, which reportedly finds the bug in minutes.
  • Opinions differ on whether static type systems or Rust‑style race‑freedom would have helped, given the likely use of unsafe/memory‑mapped IO.

General sentiment

  • Overall tone is admiration for the debugging effort, SQLite’s quality, and Tailscale’s transparency and funding strategy, tempered by skepticism about pushing SQLite in very concurrent, high‑scale, non‑standard ways.