How do HTTP servers figure out Content-Length?

Content-Length correctness & real-world bugs

• Many servers respect a manually set Content-Length and disable chunking, enabling progress bars for large downloads.
• A common bug: responses are compressed (e.g., gzip) after Content-Length is set, so the header no longer matches the body size.
• Consequences reported: clients terminate early, see truncated or garbage responses, and servers log persistent “connection dropped” warnings.
• Some hardware/web UIs and proxies historically had off‑by‑one Content-Length issues; browsers handle these inconsistently.
• Several commenters stress that Content-Length should not be trusted blindly; at best it’s a hint.

Security and HTTP desynchronization

• Mismatched lengths and bodies can, in principle, lead to HTTP request smuggling or desync, especially behind load balancers/proxies.
• Others argue that in many implementations this is more a client‑side corruption issue than a direct server memory‑safety bug, but agree behavior varies by stack.
• There’s mention of an entire class of “HTTP Desync Attacks” that exploit exactly these parsing discrepancies.

Framework behavior: buffering, chunking, streaming

• Go’s auto-chunking is contrasted with other stacks that require explicit length/chunking decisions.
• PHP is cited as buffering by default and auto‑chunking if no Content-Length is set, which can hide HTTP details from developers.
• Some frameworks still assume full in‑memory buffering; streaming large responses is described as surprisingly hard, especially with non‑blocking stacks.
• A recurring criticism: frameworks that turn all responses into strings, throwing away underlying streaming capabilities.

Compression, range requests, and progress reporting

• Compression with Content-Encoding complicates progress indication and range requests, since headers describe compressed size while APIs often expose only decompressed data.
• fetch() in browsers is criticized for lacking straightforward progress reporting on compressed streams; XMLHttpRequest is still preferred for this use case.
• Workarounds discussed include avoiding HTTP compression for large files, pre‑zipping, or embedding size metadata, each with trade‑offs.

HTTP complexity and protocol versions

• Some argue HTTP/1.0 is simple and a 1.0 server can be written quickly; others note HTTP/1.1 and especially 2/3 are significantly more complex.
• The length and detail of the specs, plus cache behavior and multiplexing, are cited as evidence that HTTP overall is not “simple.”

Chunked transfer, trailers, and alternative mechanisms

• Chunked transfer is positioned as useful for unknown‑size or never‑ending streams, distinct from WebSockets (different protocol, semantics, and deployment issues).
• WebSockets are seen as better for bidirectional realtime communication; chunked HTTP for one‑way streaming and broad compatibility.
• Some creative uses: interactive apps as a single long‑lived chunked HTML response; custom progress protocols layered over chunked encoding.
• HTTP trailers are noted as a lesser‑known feature; awareness and support in middleboxes and browsers is described as poor or partial.

Performance and implementation anecdotes

• Older “fit in one TCP packet” optimizations (e.g., 14KB pages) are mentioned as a historical tuning practice; modern TCP/QUIC and CDN settings complicate this.
• Implementers share experiences with off‑by‑one Content-Length bugs, terminating null bytes, and heuristic fixes in commercial proxies.
• Receiving requests efficiently (buffering, peeking) and managing fragmented buffers (e.g., in embedded stacks) are highlighted as non‑trivial implementation details.