We tested 20 LLMs for ideological bias, revealing distinct alignments

Methodology & Limitations

• Commenters find the prompt set narrow: few questions per axis, English-only, forced A/B/pass answers, and a single system prompt.
• Suggestions: expand question bank, add multiple phrasings per item, randomize option order, and translate prompts to see how different language “Overton windows” shift answers.
• Small sample size per axis means a model can flip from 0% to 100% in a category just by changing 1–2 answers, so per-axis claims are seen as fragile.

Nature and Sources of Bias

• Many argue bias is unavoidable: models inherit it from human data; selection of training sources (news, publishers, social platforms) skews toward elite or “progressive” online discourse rather than general populations.
• Others stress bias isn’t layered on a neutral engine; it’s baked into the weights through data and RLHF.
• Labels like “progressive” and “conservative” are criticized as narrow (heavily tied to abortion/gender/social-norm questions) and not capturing economic or geopolitical stances; several see most models as “establishment / neoliberal” rather than truly progressive.

Interpretation of Results

• Observed pattern: almost all models lean regulatory and socially progressive, with a few outliers leaning libertarian or conservative on some axes. Some call this polarization; others call it uniformity.
• Specific findings spark debate: pro-UN/NATO/EU stances, pro-Israel answers, disagreement with international law on resistance to occupation.
• Distillation and newer versions sometimes show ideological shifts, raising questions about what alignment steps changed.

Control, Neutrality, and “Truth”

• One thread: are models mostly shaped by data, or are companies intentionally steering politics? Both possibilities are considered.
• Several argue it’s impossible to define a single “neutral” position; “unbiased” might mean mirroring empirical consensus, not 50/50 on every controversy.
• Others propose LLMs should refuse direct answers on hot-button questions, instead outlining competing arguments—but even choosing which “facts” to present is itself contested.

Social & Political Implications

• Concerns that LLM-backed devices could become new chokepoints: effectively banning or monetizing disfavored activities, similar to payment-processor control.
• Fear that ideologues will game training data and alignment to capture “perceived truth,” using LLMs as a new propaganda vector.
• Counterview: LLMs are tools; healthy users should treat them as maps of discourse, not oracles.

Proposed Responses

• Add explicit political-bias sections to model cards; place models on ideological compasses, including compliance rates.
• Provide transparency about training and alignment and allow user choice among differently biased models, analogous to picking newspapers or feed algorithms.
• Accept that the real task is not “removing” bias but choosing which bias is appropriate for a given application.