AI outperforms law professors in Stanford Law study

Study scope and methodology

• Many see the headline as overreaching vs the actual claim: law professors preferred AI-tutor answers for first‑year contracts questions, not that AI “beat” professors at law practice.
• Commenters question sample size (16 instructors), high variance between instructors, and statistical power.
• Concerns about bias: models are Google’s (Gemini, NotebookLM); Stanford’s AI affiliations and big-tech funding raise perceived conflicts of interest.
• Evaluators likely graded quickly and superficially; preference tests may reward style and confidence over rigor or correctness.
• Some note the questions were open‑book, conceptual, and without single “right” answers, making preference a soft metric.

Use of AI as tutor vs. lawyer

• Strong support for AI as a tutor or “super‑paralegal”: explaining concepts, generating examples, suggesting counter‑arguments, helping students prepare better questions.
• Many argue the study does not justify using AI for direct legal counsel or high‑stakes filings.

Reliability, hallucinations, and verification

• Repeated reports of hallucinated or misused case law and statutes, even when citations look plausible.
• Knowledge cutoffs and jurisdictional gaps make models miss recent or local precedents.
• Some use RAG and legal databases (Westlaw, Lexis, court APIs) plus manual checking to mitigate errors.
• Several stress that in law, unlike code, there are no tests, logs, or easy rollbacks; errors can surface years later and be irreversible.

Domain expertise and human-in-the-loop

• Consensus: AI is powerful in expert hands and dangerous for naive users, especially pro se litigants.
• Experts can spot subtle mistakes and use AI to speed research and drafting; juniors and non‑lawyers may be misled by polished but wrong answers.

Practical use cases and risks

• Suggested uses: first-pass contract drafts, research, style/structure polishing, generating issue lists for lawyers to review.
• High concern around AI writing wills, core contracts, or briefs without expert review due to “legal footguns.”

Access to justice, economics, and jobs

• Some hope AI lowers barriers to legal knowledge and reduces costs, improving access to justice.
• Others note law’s social and relational aspects (knowing judges, local practice norms) that AI cannot easily capture.
• Debate over how far AI threatens legal jobs vs. mainly reshapes workflows and shifts value to oversight, trust, and accountability.