Weird Lexical Syntax

Original Article ↗ Hacker News Discussion ↗

Programmers trade examples of how surprisingly complex and irregular real-world programming language syntax can be, from Java’s Unicode escapes and Perl’s undecidable parsing to Ruby’s heredocs and TeX’s reprogrammable lexer. Many comments focus on the challenge this poses for tools like syntax highlighters and parsers, debating approaches such as DFA-based lexers, Tree-sitter, and full AST parsing, as well as the limits of handling nested interpolation and version-specific quirks. The thread ultimately underscores that reliably “understanding” code often requires more than simple lexical rules and sometimes even demands executing or simulating the language.

Lexing, comments, and Unicode edge cases

  • Forth: multiple string forms (.", S", C") and easily extensible comment syntaxes show you often must execute code to lex it.
  • Java: Unicode escapes are processed everywhere, even in comments, causing surprises and potential security/obfuscation risks (e.g., terminating comments via escapes).
  • C and ML: C lacks nested comments; ML/Pascal support them, which some find more ergonomic. Others argue nested comments aren’t worth the extra complexity or backward-compat issues.
  • Git commit messages: # as comment marker clashes with IDs and Markdown-style messages; workarounds include changing the comment char or cleanup mode.

Strings, quoting, and interpolation complexity

  • Many languages (C#, Python, JS, Ruby, shell, Make, Julia, Scala, SQL, PostgreSQL, Perl, Ruby heredocs, etc.) have elaborate interpolation and quoting: heredocs, dollar-quoting, raw strings, user-chosen delimiters, nested comments in strings, and even whitespace as a Ruby string delimiter.
  • Interpolation often embeds arbitrary expressions, making lexing non-regular. Some argue this pushes lexers into parser territory (pushdown automata); others show designs where the parser owns the nesting while the lexer just segments “string parts.”
  • Python f-strings evolved (nesting, quotes inside expressions), breaking some highlighters and making behavior version-dependent.

Undecidable or context-sensitive syntax

  • Perl, Bash, GNU Make, POSIX shell, C++ templates: parsing can depend on runtime info or type information, making full static parsing undecidable or context-sensitive.
  • Even C lexing can require symbol tables (lexer hack) if you want to distinguish certain constructs precisely.

Syntax highlighting strategies and limitations

  • Strong debate: DFA/regex-style highlighters (like Vim/joe) vs. full parsers or AST-based approaches.
  • Many argue full parsing is expensive and tricky on invalid code; good highlighters must work on incomplete or broken programs.
  • Some propose using LLMs or Tree-sitter for highlighting; concerns raised about dependencies (Rust, C++ package managers), coverage breadth, performance, and complexity.

Lisp, TeX, and extensible syntax

  • Lisp reader macros and TeX (and some Forths) can reprogram their own lexers, making full highlighting or static lexing impossible in general.
  • Discussion on Lisp’s “simple” syntax: some see it as reducing surface complexity; others note that macros and special forms reintroduce substantial syntactic/semantic complexity at a different layer.