We migrated our PostgreSQL database with 11 seconds downtime

Original Article ↗ Hacker News Discussion ↗

A UK government team describes migrating a 400GB PostgreSQL database for its GOV.UK Notify service to a new AWS RDS instance with just 11 seconds of downtime, using AWS Database Migration Service, DNS tricks, and logical replication. Commenters contrast this approach with alternatives such as native PostgreSQL tools and RDS blue/green deployments, noting both the power and the pitfalls of DMS, especially at scale or with complex schemas. The move also reignites a broader debate over governments’ reliance on foreign cloud providers, weighing operational resilience and staffing constraints against data sovereignty, cost, and long‑term control of critical infrastructure.

Use of AWS by UK Government

  • Many are uneasy that a core UK government service runs on AWS, a US company, with concerns about data sovereignty, geopolitical risk, and dependence on a foreign vendor.
  • Others argue AWS brings strong compliance, security, and reliability, and is likely better than typical in‑house government data centers.
  • Some note the service was already on AWS via GOV.UK PaaS; this move mainly changes accounts/architecture, not the basic dependency.

Cloud vs On‑Prem / Sovereignty Debate

  • Critics say a rich G7 country should build its own “public sector cloud” or run on‑prem to retain institutional expertise and control.
  • Counter‑arguments: government IT hiring is slow and underpaid; on‑prem is expensive, complex, and often outsourced to large integrators anyway; public cloud eases procurement and scaling (e.g. pandemics).
  • There’s broader worry about pervasive dependence on US tech (AWS, Microsoft, Palantir, etc.).

AWS DMS and Migration Approaches

  • Thread includes mixed experiences with AWS Database Migration Service (DMS):
    • Some call it buggy, opaque, and poorly supported, with reports of silent data corruption, schema change issues, and type limitations.
    • Others report successful use for MySQL↔Postgres and on‑prem→AWS, but stress careful testing and minimal transformations.
  • Several note AWS’s own docs recommend native Postgres tools (pg_dump, logical replication) for Postgres→Postgres migrations; DMS may add needless complexity.

Downtime, DNS, and Query Handling

  • The article’s 11 seconds of downtime is seen as good, but some argue further reduction is possible by pausing connections via pgbouncer/pgpool instead of hard cutover.
  • Reliance on DNS TTL=1s worries some, since not all resolvers honor TTL; connection pooling behavior is discussed.
  • Long‑running queries and transactions are highlighted as enemies of zero‑downtime cutovers; timeouts and operational discipline help.

Alternative Tools and Patterns

  • Multiple commenters advocate Postgres logical replication, pglogical, pgloader, or physical/WAL replication for low‑downtime moves.
  • RDS Blue/Green Deployments receive strong praise for near‑zero‑downtime version upgrades and even encryption changes, though limited across accounts and to forward upgrades only.

Government IT Culture and Procurement

  • Several describe public‑sector IT as constrained by budgets, rigid procurement, slow hiring, and risk‑averse culture, driving “buy, not build” decisions.
  • Some see publishing a detailed, clear migration write‑up as a positive sign of transparency and decent engineering practice.