ChatGPT does Advent of Code 2023
Advent of Code 2023 is being used as an informal benchmark for how well large language models like ChatGPT can solve novel programming puzzles, with many noting that the problems stump current models far more than earlier years. Commenters debate whether this reflects deliberate “LLM-unfriendly” puzzle design (which the creator explicitly denies), inherent limits of current architectures, or simply the absence of training data for these unique tasks. The thread broadens into questions about how much LLMs can truly aid programmers—especially at debugging and reasoning—how far scaling can take them, and whether they meaningfully change everyday development workflows rather than replace human problem‑solving.
Was AoC 2023 Deliberately Made LLM‑Resistant?
- Several commenters initially assumed the puzzles were crafted to confuse LLMs (extra conditions, tricky wording, need to inspect inputs).
- Others point out the puzzle author explicitly denied any LLM influence and described the same process as prior years.
- Some reconcile this by saying the author may unintentionally favor styles that are hard for LLMs, or that “AI‑adverse” can just mean “genuinely new problems not in the training set.”
Puzzle Characteristics & Input Idiosyncrasies
- Multiple people note more problems than usual benefitted from studying specific input structure or special cases.
- Examples include circuit‑like tasks broken into easier subgraphs and problems that become far easier than the general case once input quirks are noticed.
- Others say such input‑dependent tricks have existed in earlier years too.
LLM Performance on AoC & Coding
- Some report GPT‑4 failing even early 2023 days without heavy guidance, especially on nuanced parsing.
- Others show GPT‑4 can add debug logging, interpret outputs, and iteratively fix its own code when given an interpreter and clear guidance.
- There’s disagreement on debugging: some say its debugging skills are “non‑existent,” others say it’s strong if asked for diffs, diagnoses, or focused help rather than full rewrites.
- AoC is seen as a strong benchmark for general reasoning, but not necessarily representative of everyday business coding.
Productivity & Benchmarking Debates
- Many use LLMs to speed up boilerplate, regexes, clarifying puzzle statements, and rough scaffolding.
- A suggested real test: whether an average programmer with GPT‑4 outperforms an equally skilled one without it, and by what margin.
- Comparisons are made to other productivity factors (language choice, IDEs, syntax highlighting).
Scaling, Data, and “Peak AI” Concerns
- Some predict substantial future gains from more compute, data, and techniques (MoE, synthetic data).
- Others argue we’re nearing data limits and that LLMs are fundamentally pattern mimickers, not path to “true” intelligence or sentience.
- There’s worry about overfitting and models simply regurgitating existing AoC and tutorial code rather than reasoning from first principles.
Debugging & Education
- A side thread criticizes CS education for over‑emphasizing perfect code and under‑teaching debugging.
- AoC (and LLM use) expose how crucial debugging and reasoning about flawed code really are.