Is GPT-4 a good data analyst? (2023)

Overall view on “GPT‑4 as data analyst”

• Several commenters think the paper’s conclusion (“GPT‑4 performs comparable to a data analyst”) is overstated or effectively meaningless given the setup.
• Experiments are seen as too clean: small, well-structured tables, “SQL 101” questions, no messy real‑world schemas, incentives, or business context.
• Others argue that, even in 2023, GPT‑4 already looked “pretty good” on simple analytics tasks and that rapid model progress makes the paper quickly dated.

Understanding vs. pattern matching

• One major thread debates whether LLMs “understand” anything or just match patterns.
• Some insist transformers only learn statistical token relationships, with no grounding or self‑aware reasoning.
• Others counter that learned embeddings and concept clusters (e.g., dogs, faces, “snake‑like” websites) are a practical form of understanding.
• A recurring theme: if humans are also sophisticated pattern matchers, the distinction may be less sharp than critics assume.

Chess and reasoning as tests

• Chess is used as a testbed for understanding and reasoning.
• Some report GPT‑4 confidently making illegal moves despite correctly explaining the rules, taking this as evidence of shallow, ungrounded “knowledge.”
• Others note that some LLM variants can play at club‑level Elo, and that humans also often know rules without being able to execute them well.
• Debate centers on whether “passing behavioral tests” is sufficient to call it understanding, or just sophisticated simulation.

Real‑world data work and semantic layers

• Practitioners note GPT‑style models do fine on a single clean CSV but struggle with real warehouses: messy column names, typos, duplicated data, multiple sources (Postgres, SaaS systems).
• Semantic layers and centralized data models are proposed as necessary scaffolding; they’re hard, domain‑specific, and largely manual.
• Some companies are building tools to combine LLMs with semantic layers or prompt “engines” to autonomously explore databases and generate reports.

Hallucinations, prompts, and reliability

• LLMs frequently hallucinate details about research papers and citations; even when corrected, they may simply generate the next likely apology.
• One view: treat LLMs as “blurry JPEGs of the internet” or “habitual liars / expert beginners” and verify everything.
• Prompt quality matters: right prompts can yield strong SQL and analysis on toy tasks, but users rarely craft perfect prompts.
• Many see LLMs as highly useful assistants yet fundamentally unsafe as sole sources of truth or for tasks demanding guaranteed correctness.