Unexpected security footguns in Go's parsers

Original Article ↗ Hacker News Discussion ↗

Surprising parser behaviors & polyglot payloads

  • Many were surprised that a single payload can be valid JSON/YAML/XML and that Go’s XML decoder accepts leading/trailing garbage while still producing a “valid” struct.
  • This is seen as classic “parser differential” material: multiple components see the “same” input differently, which can be exploitable.
  • Similar issues exist elsewhere (e.g., Python’s JSON parser hitting RecursionError on deep invalid input, contrary to docs).

Go JSON design choices and security implications

  • Case‑insensitive key matching in Go’s JSON unmarshaler is widely criticized as “insane” and a clear footgun, especially since most other languages treat keys case‑sensitively.
  • Default behavior of serializing all exported struct fields and assuming loose input (unknown fields, trailing garbage with streaming) is viewed as favoring convenience over safety.
  • Some defend these as pragmatic 80/20 design: simple for common cases, with complexity pushed to edge cases. Others argue these “simplifications” cause predictable, serious bugs.

Struct tags and stringly‑typed metadata

  • Heavy debate over Go’s struct tags (json:"...,omitempty") as a “hidden DSL in strings”:
    • Critics: brittle, hard to validate, inconsistent conventions between libraries (json, gorm, etc.), easy to mis‑type options (- vs -,omitempty).
    • Defenders: far simpler than Java annotations or macros, enough for 80% of needs, keeps metaprogramming “magic” low.
  • Comparison with Rust macros, Java/.NET attributes, F# type providers, OCaml PPX, etc., which offer safer, structured metadata but at higher conceptual cost.

Visibility, casing, and unintended exposure

  • Go’s public/private semantics tied to capitalization mean JSON keys often differ (User vs user), motivating the case‑insensitive behavior.
  • Some suggest keeping sensitive fields unexported or using json:"-", but that can conflict with ORMs (e.g., private fields skipped) and cross‑package access.
  • Several argue that tightly coupling DB models and API structs is the deeper problem, as it leads to accidental leaks and hard‑to‑change APIs.

DTO separation, ORMs, and “fat” vs “narrow” structs

  • Strong camp: always separate DTOs (request/response types) from domain/storage models to avoid over‑exposing fields and to make refactoring safe.
  • Counterpoint: proliferation of narrow structs plus mapping code feels like boilerplate; some prefer “fat” structs and manual parsing of generic JSON trees instead of annotation‑based unmarshaling.
  • Others note that modern mapping tools (e.g., MapStruct‑like libraries) can automate DTO↔model copying, though Go culture tends to resist such complexity.

Parsers vs validation / authorization layers

  • One view: “there are no footguns;” parsers should just parse. Security requires explicit validation/whitelisting and constructing new, validated structures or re‑serializing trusted data between components.
  • Another view: defaults still matter; permissive parsers and surprising behaviors (case‑insensitivity, garbage‑tolerant XML) materially increase the chance of developer mistakes in real systems.
  • For SAML/XML‑signature cases, some emphasize ensuring the processing layer operates only on the authenticated bytes, not on the original input.

Duplicate keys, unknown fields, and versioning

  • Discussion around how to handle duplicate JSON keys: “last wins,” “first wins,” error, or nondeterministic. Consensus: there is no perfect answer; any choice can cause differentials.
  • Some support the article’s suggestion to standardize on “last wins” because it’s most common; others say the real fix is ensuring the same parser/semantics are used across boundaries.
  • DisallowUnknownFields is debated:
    • Pros: catches mistakes and useless/rogue fields early.
    • Cons: makes forward/backward compatibility harder; some advocate strict, versioned APIs instead (e.g., /api/v1, /api/v2) and exact parsing per version.

Alternative formats and schemas (Protobuf, OpenAPI, etc.)

  • A few see this as an argument for Protocol Buffers or schema‑first OpenAPI with codegen, to get more consistent ser/de and stricter typing.
  • Others push back: Protobufs still inherit language differences (ints, strings, etc.) and don’t eliminate parsing/semantics disputes; they just move them.
  • Several suggest using dedicated validation/parsing layers (e.g., zod in TypeScript, strict JSON schemas) and possibly re‑encoding data at trust boundaries.

Is this uniquely Go?

  • Some argue the article over‑targets Go and is “clickbaity”; these issues (duplicate keys, flexible decoding, struct auto‑mapping) exist in many ecosystems.
  • Others respond that Go’s specific defaults—case‑insensitive JSON keys, automatic serialization of all exported fields, lax XML—are genuine, distinctive footguns that have already produced real CVEs.
  • Broad agreement: JSON/XML are messier and more dangerous in practice than their surface simplicity suggests; secure design requires explicit boundaries, validation, and careful API/model separation, regardless of language.