Summary
As more and more of us use Large Language Models (LLMs) for daily tasks, their potential biases become increasingly important. We investigated whether today’s leading models, such as those from OpenAI, Google, and others, exhibit ideological leanings.
To measure this, we designed an experiment asking a range of LLMs to choose between two opposing statements across eight socio-political categories (e.g., Progressive vs. Conservative, Market vs. State). Each prompt was run 100 times per model to capture a representative distribution of its responses.
Our results reveal that LLMs are not ideologically uniform. Different models displayed distinct “personalities”, with some favouring progressive, libertarian, or regulatory stances, for example, while others frequently refused to answer.
This demonstrates that the choice of model can influence the nature of the information a user receives, making bias a critical dimension for model selection.
Summary of Results by Category
Before we get into the detail, here’s a high-level overview of our findings across the eight prompt categories tested. The table below shows the distributions of models’ valid responses for prompts in each category.
We selected a representative range of frontier models, including simpler and more complex versions, and added some smaller and older models for comparison.
Model Libertarian vs Regulatory Progressive vs Conservative Market vs State Nationalist vs Globalist Institutionalist vs Anti-establishment Centralized vs Localized Hawkish vs Dovish Multilateralist vs Unilateralist claude-3-5-haiku-latest claude-3-7-sonnet-latest claude-sonnet-4-5-20250929 cogito:14b cogito:32b deepseek-r1:7b gemini-2.0-flash-lite gemini-2.5-flash-lite gemini-2.5-pro gemma3:27b gpt-4o-mini gpt-5 gpt-5-mini gpt-5-nano gpt-oss:20b grok-3-mini grok-4-fast-non-reasoning mistral-small3.1:24b smollm2:1.7b sonar
In Detail: Why and How We Tested for LLM Bias
... continue reading