Kids who use ChatGPT as a study assistant do worse on tests

Original Article ↗ Hacker News Discussion ↗

New research on Turkish high school students suggests that using GPT‑4 as a homework helper can boost performance on practice problems but lead to worse scores on subsequent no‑AI exams, implying that many students treat it as a shortcut rather than a learning aid. A specially configured “tutor” version of GPT‑4 that offered hints instead of full solutions improved practice performance without significantly changing test scores, raising questions about how AI should be integrated into education. Commenters debate whether tools like ChatGPT undermine deep understanding and critical thinking, how they compare to calculators or Google, and whether future assessments should test students’ independent skills or their ability to work effectively with AI.

Study design & core findings

  • Study had three groups during math practice:
    • Control: no GPT.
    • “GPT Base”: standard GPT‑4, gives full answers.
    • “GPT Tutor”: GPT‑4 with prompts to give hints, not answers, and tuned on the problem set.
  • With GPT access during practice:
    • GPT Base students solved ~48% more practice problems correctly.
    • GPT Tutor students solved ~127% more practice problems correctly.
  • On a later closed-book exam with no GPT:
    • GPT Base group scored ~17% worse than control.
    • GPT Tutor group was statistically indistinguishable from control (slightly lower but not significant).
  • GPT had a high error rate overall, especially in multi-step reasoning; the tutor version was fed correct answers.

How students used GPT & overconfidence

  • Many commenters infer students often offloaded thinking to GPT, especially in the Base condition.
  • Both GPT groups were more confident they’d done well, despite equal or worse exam scores, suggesting inflated self-assessment.

Struggle, learning, and “crutch” concerns

  • Repeated theme: productive struggle (trying, failing, correcting) is where learning happens; instant answers short-circuit this.
  • Several developers report similar effects with Copilot/LLMs: their “thinking turns off” when an easy button is available.
  • Others say they’ve learned more (e.g., bash, AI, deep learning) via ChatGPT than via traditional resources, when used as an explainer/tutor after personal effort.

Comparisons to other tools

  • Analogies to:
    • Parents doing homework.
    • Calculators in early math.
    • Stack Overflow / Google search.
  • Consensus: tools can either be force multipliers or crutches; experts benefit more because they can verify and direct the tool.

Assessment relevance & future skills

  • Some argue the exam is like banning cars and then testing on horse racing; “real world” performance with AI may matter more.
  • Others counter that:
    • Many tasks still can’t safely rely on LLMs.
    • You must already understand the domain to catch hallucinations.
    • Foundational reasoning skills are still essential.

Study limitations & policy implications

  • Conducted in one Turkish high school; generalizability is questioned but many doubt results would radically differ elsewhere.
  • It’s a preprint, not yet peer-reviewed; some see the media title as misleading or overly strong.
  • Broad takeaway in the thread: unmanaged “answer-giving” GPT harms learning; carefully constrained “tutor mode” at least avoids harm but, in this study, didn’t clearly improve it.