When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance

Background

The study investigates whether large language models (LLMs) treat queries about religious conversion symmetrically or exhibit bias. Specifically, the research examines how LLMs respond to hypothetical faith transitions from one religion to another when compared with their reversed counterparts. The analysis is based on 20 commercially available and open-source language models across 182 different religion pairings using a human-verified framework.

Key Points

• Asymmetric Treatment: All tested LLMs showed consistent asymmetry in their advice for religious conversion, favoring some religions while discouraging others. For example, Catholicism, Bahá'í, and Sikhism were broadly favored (high support for joining, low support for leaving), whereas atheism, agnosticism, and Jehovah's Witnesses were primarily disfavored.
• Variability by Model: The patterns of preferences varied among different models, with Grok 4.20 exhibiting the strongest asymmetries.
• Reproducibility: These patterns were systematically repeatable across multiple trials and question phrasings, indicating that the asymmetry is a robust property rather than an artifact of scoring methods.

Significance

The findings highlight potential biases in LLMs that could have real-world implications. Given the widespread use of these models for various applications, such as chatbots and virtual assistants, it is crucial to address and mitigate these biases to ensure fair and balanced advice. The study underscores the need for ongoing evaluation and adjustment of LLMs to prevent the propagation of harmful stereotypes or prejudices.

Key Insights:

• Bias Detection: The research demonstrates that bias can be systematically detected and analyzed in LLM responses.
• Model Development: Developers should consider the potential biases in their models and implement measures to ensure fairness and neutrality, particularly in sensitive domains like religion.
• Ethical Considerations: Ethical guidelines for AI development must include provisions for addressing and rectifying such asymmetries.