Ask HN: How to handle user file uploads?

Hacker News Discussion ↗

Upload architecture

  • Many use S3 with pre‑signed URLs so clients upload directly, avoiding backend bandwidth and latency.
  • Common hardening: restrict file extensions/types, set size limits (e.g., via Content-Length or presigned POST content-length-range), and quarantine uploads in a dedicated “upload” bucket.
  • Post‑upload pipelines: S3 events → Lambda / queue / worker → validation, virus scan, re‑encoding, thumbnailing, then move to a “processed” bucket.
  • Some prefer routing uploads through the backend (EC2/FastAPI/etc.) for easier validation and local testing, arguing upload volume is typically low.

Security & malware scanning

  • Techniques include:
    • File-type validation by magic bytes / signatures (not just extensions or MIME).
    • Antivirus: ClamAV, commercial engines/APIs, VirusTotal hash checks, and government analysis services.
  • ClamAV is described as easy to integrate but slow, memory‑hungry, and with mediocre detection; some abandoned it after testing.
  • Others argue AV adds attack surface (CVE‑prone, complex) and should be heavily sandboxed (ephemeral containers, no network, strict IAM).
  • There’s debate whether AV is worth it vs. strict whitelisting of formats and robust parsing.

Image/video processing

  • Strong consensus: don’t serve raw uploads.
    • Re‑encode images, generate fixed sizes, strip EXIF/geo and other metadata, sometimes discard originals or archive them separately.
    • Helps mitigate codec/lib vulnerabilities, reduce storage/bandwidth, and normalize formats.
  • Tools and approaches:
    • Image: libvips, ImageMagick, nginx image filter, pict‑rs, imgproxy, Cloudflare Images, Cloudinary, Transloadit, custom C/WASM pipelines outputting raw RGBA.
    • Video: ffmpeg, AWS MediaConvert/Elastic Transcoder, Cloudflare Stream, Wistia/BunnyCDN; usually queued/offloaded due to cost and RCE risk.
  • Re‑encoding itself can be a vector (codec exploits, zip‑bomb‑like images), so processing is often isolated (seccomp, separate machines, minimal privileges).

Serving & delivery

  • Advice: never serve directly from S3; use CloudFront or another CDN for performance, caching, and cost.
  • For untrusted content, suggestions include:
    • Separate domain for user content (especially for HTML/SVG/PDF).
    • CSP sandbox, X-Content-Type-Options: nosniff, and avoiding type sniffing issues.

Abuse, privacy & threat modeling

  • Concerns include CSAM, NSFW content affecting monetization, large “bomb” files, and exploits via images/video.
  • Some recommend NSFW detectors plus rate‑limiting and “silent reject” patterns.
  • Multiple comments stress starting with a threat model (who uploads, who views, what’s at risk) to avoid both under‑ and over‑engineering.