ChatGPT is biased against resumes with credentials that imply a disability
Research showing that ChatGPT tends to rate résumés suggesting a disability lower than otherwise identical ones has renewed concern about using large language models in hiring. Commenters argue that AI systems inevitably mirror existing human and societal biases in their training data, making “neutral” or “fair” automated screening extremely difficult without heavy-handed prompt engineering that can introduce new distortions. The thread also raises legal, ethical, and practical questions about delegating high‑stakes decisions like hiring, housing, or child protection to opaque models whose criteria can’t easily be inspected or explained.
Scope of the Bias Finding
- Commenters see ChatGPT’s ableist behavior as unsurprising, paralleling known racial and gender biases in AI.
- Some test gender bias ad hoc (changing names/pronouns on identical CVs) and report no difference, but others point out this doesn’t test for cultural or name-based bias.
- Many emphasize that “don’t be biased” prompts may simply flip the bias or overweight disability signals rather than truly neutralize them.
AI as a Mirror of Human Bias
- Strong theme: LLMs reflect patterns in human data; if society and hiring are biased, the model will be too.
- Some argue this is “correct” in a statistical sense (it models reality), while others stress that “correct” for prediction is not “acceptable” for hiring or ethics.
- Several note this makes AI a powerful mirror of unconscious prejudice, but also a tempting way for organizations to outsource and deniably automate discrimination.
Technical and Methodological Limits
- Discussion of how instructions like “focus on disability justice” can swing rankings too far the other way, creating a new distortion.
- Multiple comments highlight that LLMs are black boxes, not transparent decision systems, and that deep learning makes detailed explanations of decisions essentially impossible.
- Others suggest limiting AI to narrow tasks (e.g., extracting structured data) and feeding that into explicit, auditable rules instead of free-form ranking.
Legal and Policy Concerns
- Debate over whether disparate impact (biased outcomes without explicit intent) is enough for legal liability; some say it’s hard to win in court, others note AI makes bias easier to empirically demonstrate.
- Concern that agencies and companies will adopt these tools quickly (e.g., child protection, housing, hiring) without understanding or controlling bias.
Disability, Productivity, and Hiring Practices
- Some question whether anti-discrimination laws “fight reality” if some disabilities affect productivity; others respond that reasonable accommodations often neutralize impact.
- Anecdotes: people omitting disability-related achievements or even “American Sign Language” from CVs due to suspected screening penalties.
- Several disabled or blind developers describe the dilemma of when to disclose and how little resumes convey about actual capability.