Is GPT-4 a good data analyst? (2023)
Claims that GPT‑4 can perform comparably to a human data analyst prompt scrutiny of what “understanding” actually means for large language models and whether pattern matching over text can truly substitute for reasoning. Commenters highlight both impressive feats—like generating plausible code or “snake‑like” websites—and serious weaknesses, including hallucinated research citations, illegal chess moves despite knowing the rules, and brittle performance on messy real‑world datasets. Many see LLMs as powerful but unreliable tools that work best when tightly constrained and paired with robust data modeling, with the open question being how far improved prompting, architectures, and environments can close the gap to genuine analytical reliability.
Overall view on “GPT‑4 as data analyst”
- Several commenters think the paper’s conclusion (“GPT‑4 performs comparable to a data analyst”) is overstated or effectively meaningless given the setup.
- Experiments are seen as too clean: small, well-structured tables, “SQL 101” questions, no messy real‑world schemas, incentives, or business context.
- Others argue that, even in 2023, GPT‑4 already looked “pretty good” on simple analytics tasks and that rapid model progress makes the paper quickly dated.
Understanding vs. pattern matching
- One major thread debates whether LLMs “understand” anything or just match patterns.
- Some insist transformers only learn statistical token relationships, with no grounding or self‑aware reasoning.
- Others counter that learned embeddings and concept clusters (e.g., dogs, faces, “snake‑like” websites) are a practical form of understanding.
- A recurring theme: if humans are also sophisticated pattern matchers, the distinction may be less sharp than critics assume.
Chess and reasoning as tests
- Chess is used as a testbed for understanding and reasoning.
- Some report GPT‑4 confidently making illegal moves despite correctly explaining the rules, taking this as evidence of shallow, ungrounded “knowledge.”
- Others note that some LLM variants can play at club‑level Elo, and that humans also often know rules without being able to execute them well.
- Debate centers on whether “passing behavioral tests” is sufficient to call it understanding, or just sophisticated simulation.
Real‑world data work and semantic layers
- Practitioners note GPT‑style models do fine on a single clean CSV but struggle with real warehouses: messy column names, typos, duplicated data, multiple sources (Postgres, SaaS systems).
- Semantic layers and centralized data models are proposed as necessary scaffolding; they’re hard, domain‑specific, and largely manual.
- Some companies are building tools to combine LLMs with semantic layers or prompt “engines” to autonomously explore databases and generate reports.
Hallucinations, prompts, and reliability
- LLMs frequently hallucinate details about research papers and citations; even when corrected, they may simply generate the next likely apology.
- One view: treat LLMs as “blurry JPEGs of the internet” or “habitual liars / expert beginners” and verify everything.
- Prompt quality matters: right prompts can yield strong SQL and analysis on toy tasks, but users rarely craft perfect prompts.
- Many see LLMs as highly useful assistants yet fundamentally unsafe as sole sources of truth or for tasks demanding guaranteed correctness.