Why you shouldn't leave model selection on default in Copilot, Gemini and other AI tools
Mathematician Adam Kucharski demonstrated that Microsoft Copilot fabricates country-based stereotypes when analyzing identical datasets labeled with d
Analysis
Microsoft Copilot doesn't just analyze data — it projects preconceived narratives onto it, and a simple experiment by mathematician Adam Kucharski exposed just how easily that happens.
The Experiment and What It Reveals
Kucharski's methodology was elegant in its simplicity: feed Copilot the same underlying data multiple times, but swap out the country labels each run. A genuinely analytical tool should return near-identical outputs, since the actual data didn't change. Instead, Copilot produced wildly different — and culturally stereotypical — interpretations for each country. The tool essentially hallucinated national character where the numbers supported none.
This isn't a minor calibration issue. It strikes at the heart of what users expect from an analytical assistant: objectivity. When a business analyst asks Copilot to compare markets or when a policy researcher evaluates cross-country metrics, they're trusting that the tool processes the numbers, not its training-time associations about what "Country X" is supposedly like. That trust is misplaced.
Why This Happens
The underlying problem is predictable to anyone familiar with large language models. Copilot, like all LLM-based tools, is built on patterns absorbed from massive text corpora. Those corpora contain decades of cultural commentary, news articles, and informal discourse that heavily associate certain traits with certain nations. When the model encounters a country label, it doesn't treat it as a neutral variable — it activates a web of latent associations that bleed into the output.
What makes this particularly dangerous in an analytical context is the confidence framing. Copilot doesn't hedge or flag its stereotyping. It presents fabricated cultural conclusions with the same polished certainty it would use for a mathematically derived result. Users without domain expertise have no signal that the output is contaminated.
The Thinking Model Caveat
The article notes that thinking models (reasoning-enhanced variants) can catch this distortion — but only when users actively select them. This is a critical detail. The default model configuration doesn't self-correct for stereotypical bias, meaning the vast majority of users, who never change default settings, will receive biased outputs without knowing it.
This creates a two-tier system: technically sophisticated users who know to switch models get accurate analysis, while everyone else gets dressed-up prejudice. For a product marketed as democratizing data analysis, that's an ironic failure mode. Microsoft is essentially offloading quality control to the user's own expertise — the very thing the tool is supposed to replace.
Competitive and Industry Implications
This finding doesn't exist in isolation. Google's Gemini faced its own controversy when it overcorrected on diversity to the point of generating historically inaccurate images. The pattern across the industry is clear: foundation models encode cultural assumptions, and neither under-correction (Copilot's stereotyping) nor over-correction (Gemini's earlier debacle) is acceptable for professional use cases.
For enterprises evaluating AI-assisted analytics tools, Kucharski's experiment is a reproducible litmus test. Any organization deploying Copilot for cross-regional data work — international finance, global health, comparative policy — should be running similar sanity checks. The fact that a user has to know to do this manually is itself a product design failure.
The Real Problem: Default Behavior
The article's title points to the sharpest insight: model selection defaults matter enormously. Most users interact with AI tools without understanding the architectural choices behind a dropdown menu. When the default mode produces biased analytical results, the product is systematically misinforming its largest user base. Microsoft could address this by either making the thinking model the default for analytical tasks or by adding explicit bias warnings when country-labeled data is detected. Neither is technically difficult. The fact that neither is implemented suggests the company hasn't prioritized analytical reliability as highly as it should.
Kucharski's experiment is a reminder that AI tools don't passively report reality — they actively construct interpretations. When those interpretations are shaped by stereotypes rather than data, the tool isn't analyzing anything. It's storytelling with numbers on top.
Disclaimer: The above content is generated by AI and is for reference only.