New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
An AI-driven quiz and autograding platform used in a Dartmouth statistics course reportedly boosted exam performance by up to 0.7–1.3 standard deviations and achieved 90% voluntary uptake among students who rarely read the textbook. Commenters see strong potential for scalable, personalized practice and feedback, but highlight serious concerns about selection bias, novelty effects, weak experimental design, overclaiming “tutor” capabilities, and the broader track record of ed-tech that improves engagement without clearly improving learning. Many argue that AI may meaningfully narrow the gap between classroom teaching and one‑on‑one tutoring, while stressing the need for rigorous randomized trials, careful integration into curricula, and safeguards against overreliance and hallucinations.
Study Design, Effect Size & Bias
- Reported effect size (0.71–1.30 SD) is considered large and exciting, but many comments stress strong risks of:
- Self‑selection (motivated students doing more quizzes).
- Lack of randomized control group.
- Possible novelty / Hawthorne effects.
- Potential overlap between quiz content and exam questions.
- Some argue calling this an “effect size” is misleading without a proper RCT; correlation may just show that students who engage more learn more, regardless of tool.
- A project member replies:
- Effect is estimated via regression across the full usage range, not just “super users.”
- Median lesson coverage was >90% even counting non‑users.
- Engagement persisted across the term; retries were often spaced days apart, suggesting more than novelty.
- Exams and curriculum were designed independently; the platform repackaged textbook material.
- An MCQ‑only module showed similar engagement but no dosage relationship, suggesting constructed‑response plus AI grading mattered.
- Ethics concerns limited RCTs; future crossover or partial‑feature randomization is proposed.
What the Tool Actually Is
- Several note it’s more a quiz platform with AI autograding of constructed responses than a full conversational “tutor.”
- Chat/RAG assistant features were reportedly used little.
- Debate over whether grading plus feedback counts as “tutoring” versus deeper, interactive explanation.
Engagement & Student Behavior
- Baseline textbook reading was ~10–15%; the platform reached ~90% voluntary adoption.
- Many see this as the headline: even if per-hour learning is similar or slightly worse than textbooks, far more students are actually engaging with the material.
- Others counter that better‑designed question sets alone (AI or not) could explain gains.
Broader AI-in-Education Views
- Some view this as a promising step toward narrowing the “two‑sigma” gap from individualized tutoring.
- Others worry:
- About hallucinations, especially in subjective domains.
- That AI mainly benefits already‑motivated students.
- That tying the tool to grading may push students to use LLMs to shortcut work.
Edtech & UX Context
- Educators in the thread express deep skepticism due to previous bad tech (e.g., LMS/classroom tools forced by administrators with poor outcomes).
- There’s substantial interest in:
- Low‑distraction hardware (e‑ink tablets, smart pens) feeding into LLM tutors.
- Built‑in spaced repetition and automatic generation of practice questions/flashcards, to reduce friction versus current SRS tools.