New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

Original Article ↗ Hacker News Discussion ↗

Study Design, Effect Size & Bias

  • Reported effect size (0.71–1.30 SD) is considered large and exciting, but many comments stress strong risks of:
    • Self‑selection (motivated students doing more quizzes).
    • Lack of randomized control group.
    • Possible novelty / Hawthorne effects.
    • Potential overlap between quiz content and exam questions.
  • Some argue calling this an “effect size” is misleading without a proper RCT; correlation may just show that students who engage more learn more, regardless of tool.
  • A project member replies:
    • Effect is estimated via regression across the full usage range, not just “super users.”
    • Median lesson coverage was >90% even counting non‑users.
    • Engagement persisted across the term; retries were often spaced days apart, suggesting more than novelty.
    • Exams and curriculum were designed independently; the platform repackaged textbook material.
    • An MCQ‑only module showed similar engagement but no dosage relationship, suggesting constructed‑response plus AI grading mattered.
    • Ethics concerns limited RCTs; future crossover or partial‑feature randomization is proposed.

What the Tool Actually Is

  • Several note it’s more a quiz platform with AI autograding of constructed responses than a full conversational “tutor.”
  • Chat/RAG assistant features were reportedly used little.
  • Debate over whether grading plus feedback counts as “tutoring” versus deeper, interactive explanation.

Engagement & Student Behavior

  • Baseline textbook reading was ~10–15%; the platform reached ~90% voluntary adoption.
  • Many see this as the headline: even if per-hour learning is similar or slightly worse than textbooks, far more students are actually engaging with the material.
  • Others counter that better‑designed question sets alone (AI or not) could explain gains.

Broader AI-in-Education Views

  • Some view this as a promising step toward narrowing the “two‑sigma” gap from individualized tutoring.
  • Others worry:
    • About hallucinations, especially in subjective domains.
    • That AI mainly benefits already‑motivated students.
    • That tying the tool to grading may push students to use LLMs to shortcut work.

Edtech & UX Context

  • Educators in the thread express deep skepticism due to previous bad tech (e.g., LMS/classroom tools forced by administrators with poor outcomes).
  • There’s substantial interest in:
    • Low‑distraction hardware (e‑ink tablets, smart pens) feeding into LLM tutors.
    • Built‑in spaced repetition and automatic generation of practice questions/flashcards, to reduce friction versus current SRS tools.