Amazon S3 Adds Put-If-Match (Compare-and-Swap)

Original Article ↗ Hacker News Discussion ↗

Amazon S3 has introduced conditional write support (“Put-If-Match”), allowing uploads to succeed only if an object’s ETag matches an expected value, effectively adding a compare-and-swap primitive to the service. Commenters note this brings S3 in line with long‑standing features in Google Cloud Storage, Azure, and MinIO, and makes it far easier to implement optimistic concurrency, locks, and even database‑like systems directly on top of object storage without extra coordination services. Much of the debate centers on how this interacts with S3’s consistency guarantees, the limitations of ETag and checksum handling (especially with multipart uploads), and the broader trend of using object stores as the foundation for higher-level data platforms.

Significance of S3 Put-If-Match (CAS)

  • Seen as the missing primitive to safely coordinate multiple writers to the same object (e.g., WALs, logs).
  • Combined with strong read-after-write consistency, it enables optimistic concurrency control directly on S3.
  • Many note that GCP, Azure, and MinIO have had similar conditional write / ETag-based controls for years; some are surprised it took S3 so long.
  • Some question why “standard ETag support” is headline-worthy, others argue the scale and engineering complexity at S3 make the delay understandable.

Use Cases and New Patterns

  • Building databases and consensus systems directly on object storage (e.g., object-backed DBs, transactional logs).
  • Terraform-style state locking without an additional store like DynamoDB.
  • Lock-free-like patterns: repeatedly read–modify–CAS to maintain invariants (inventory counters, transaction logs).
  • Potential for “S3 as a database” or even SQLite-over-S3; acknowledged as likely slow without caching, but now technically feasible and more “serverless.”
  • Some see this reducing the need for separate coordination systems in S3-backed applications; others still mention systems like Delta Lake or external consensus services.

ETags, Hashing, and Integrity Concerns

  • Discussion of MD5-based ETags: fine for random bit-error detection, unsafe against adversarial collisions.
  • Hypothetical attacks where an untrusted party crafts MD5-colliding data to cause log entries to be “lost” if CAS is keyed only on MD5.
  • Google Cloud’s generation numbers and Azure’s concurrency controls are cited as clearer, monotonic versioning primitives.
  • Desire for content-addressable storage on S3: enforce that object key equals a secure hash of content via IAM/policies.
  • Current S3 checksums (SHA-256 and multipart “hash-of-hashes”) help with integrity but are awkward for content-addressable or CAS semantics, especially with multipart uploads.

Limitations, Ambiguities, and CAP Discussion

  • CAS and strong consistency are per-object; coordinating multi-object updates still “requires creativity.”
  • Some light debate on how this interacts with CAP: consensus that availability must occasionally be sacrificed to preserve consistency under partitions.
  • Unclear exactly when comparisons happen for large uploads (early vs final commit), and whether there are performance impacts on other operations.

Ecosystem and Meta Points

  • Open-source and proprietary systems built on object stores plan to exploit this immediately.
  • Some push back on promotional comments about commercial products; others see them as valid technical case studies.