AI outperforms law professors in Stanford Law study
A Stanford Law study reports that AI systems like Google’s Gemini produced answers to first‑year contract law questions that professors preferred over those written by other professors in 75% of blind comparisons. Commenters question the study’s methodology, potential funding bias, and reliance on subjective “preference” rather than factual correctness, especially given issues like hallucinated case citations and outdated knowledge. Many see near‑term value in AI as a tutoring and research aid or “paralegal,” but warn that high‑stakes legal work still demands human expertise, accountability, and deep, jurisdiction‑specific judgment.
Study scope and methodology
- Many see the headline as overreaching vs the actual claim: law professors preferred AI-tutor answers for first‑year contracts questions, not that AI “beat” professors at law practice.
- Commenters question sample size (16 instructors), high variance between instructors, and statistical power.
- Concerns about bias: models are Google’s (Gemini, NotebookLM); Stanford’s AI affiliations and big-tech funding raise perceived conflicts of interest.
- Evaluators likely graded quickly and superficially; preference tests may reward style and confidence over rigor or correctness.
- Some note the questions were open‑book, conceptual, and without single “right” answers, making preference a soft metric.
Use of AI as tutor vs. lawyer
- Strong support for AI as a tutor or “super‑paralegal”: explaining concepts, generating examples, suggesting counter‑arguments, helping students prepare better questions.
- Many argue the study does not justify using AI for direct legal counsel or high‑stakes filings.
Reliability, hallucinations, and verification
- Repeated reports of hallucinated or misused case law and statutes, even when citations look plausible.
- Knowledge cutoffs and jurisdictional gaps make models miss recent or local precedents.
- Some use RAG and legal databases (Westlaw, Lexis, court APIs) plus manual checking to mitigate errors.
- Several stress that in law, unlike code, there are no tests, logs, or easy rollbacks; errors can surface years later and be irreversible.
Domain expertise and human-in-the-loop
- Consensus: AI is powerful in expert hands and dangerous for naive users, especially pro se litigants.
- Experts can spot subtle mistakes and use AI to speed research and drafting; juniors and non‑lawyers may be misled by polished but wrong answers.
Practical use cases and risks
- Suggested uses: first-pass contract drafts, research, style/structure polishing, generating issue lists for lawyers to review.
- High concern around AI writing wills, core contracts, or briefs without expert review due to “legal footguns.”
Access to justice, economics, and jobs
- Some hope AI lowers barriers to legal knowledge and reduces costs, improving access to justice.
- Others note law’s social and relational aspects (knowing judges, local practice norms) that AI cannot easily capture.
- Debate over how far AI threatens legal jobs vs. mainly reshapes workflows and shifts value to oversight, trust, and accountability.