AI advice made people less accurate but more confident – sudy
AI-powered advice appears to make people more confident while less accurate, according to a study where access to an LLM greatly reduced participants’ willingness to say “I don’t know” even when the model was often wrong. Commenters question the study design—arguing similar effects might arise from any authoritative-seeming but flawed source—yet broadly agree it highlights automation bias and over-trust in AI. Many point to real-world examples like Reddit, workplace chat, and technical forums where users paste unvetted AI outputs as their own, raising concerns about degraded information quality, lost critical thinking, and the need for better norms and model behaviors.
Study link and basic findings
- Commenters tracked down the actual preprint and shared it; some preferred reading it directly rather than secondary coverage.
- Headline summary repeated: AI advice reduced “I don’t know” responses and overall accuracy, while boosting confidence.
- Some noted participants were financially incentivized to be correct, yet still followed bad AI answers.
Critiques of study design and scope
- Many argue the experiment doesn’t isolate anything uniquely “AI”: it’s essentially “open-book with a faulty reference.”
- The model used (Step 3.5 Flash) was deliberately chosen and prompted to be wrong on obscure movie-visual trivia; critics say this is unlike normal AI use and heavily biases results.
- Missing control: no comparison to non-AI tools (e.g., a bad textbook or bad search results), so the effect might be generic “automation bias.”
- Others counter that this is still relevant: LLMs are in fact unreliable on many factual questions, and the finding “people over-trust authoritative-looking but wrong tools” is important regardless.
Questions about realism and stakes
- The questions were trivial film details; several people doubt that behavior on low-stakes trivia maps to important real-world decisions.
- Others suggest higher monetary stakes or clearer warnings (“this AI is often wrong here”) would change behavior and should be tested.
Anthropomorphism, sycophancy, and product design
- Discussion of how marketing, UI, and “polite, confident” style encourage people to treat LLMs like smart coworkers rather than fallible tools.
- Debate over whether humans are inevitably prone to anthropomorphize, or whether design and culture can meaningfully mitigate this.
Impact on online advice and culture
- Strong consensus that advice/information subreddits and some forums are flooded with unvetted LLM copy-paste, degrading quality.
- People describe workplace and community norms emerging: discouraging raw AI dumps, requiring personal synthesis, and treating unlabeled AI text as dishonest or disrespectful.
- Concerns that AI-generated slop feeds back into training data, causing a downward spiral in information quality.
Broader worries and proposed responses
- Fears of “AI-assisted Dunning–Kruger”: less knowledge, more confidence.
- Suggestions: avoid using LLMs in safety-critical contexts, keep experts in the loop, emphasize grounding and verification, and teach “critique the AI” as a skill.