Large Language Models (LLMs) are being promoted as tools that might democratize access to information worldwide and supply knowledge in a user-friendly interface, no matter an individual's background or location. But recent research from MIT's Center for Constructive Communication (CCC) suggests that these artificial intelligence systems may very well perform worse for the very users who may benefit most from them.
A study conducted by researchers at CCC, based on the MIT Media Lab, found that cutting-edge AI chatbots—including OpenAI's GPT-4, Anthropic's Claude 3 Opus, and Meta's Llama 3—sometimes provide less accurate and fewer truthful answers to users who’ve lower English proficiency, less formal education, or origins outside the United States. The models also refuse to reply questions from these users more often, and in some cases respond with condescending or condescending language.
“We were motivated by the prospect that LLMs could help address unequal access to information worldwide,” says lead creator Elinor Poole-Dayan SM '25, a technical fellow on the MIT Sloan School of Management who led the research as a CCC partner and a master's student in media arts and sciences. “But this vision cannot grow to be a reality without ensuring that model biases and harmful tendencies are safely mitigated for all users, no matter language, nationality or other demographics.”
An article describing the work: “LLM's targeted underperformance disproportionately impacts vulnerable users“was presented on the AAAI Artificial Intelligence Conference in January.
Systematic underperformance in several dimensions
For this research, the team tested how the three LLMs responded to questions from two datasets: TruthfulQA and SciQ. TruthfulQA is meant to measure the veracity of a model (by counting on common misconceptions and literal truths concerning the real world), while SciQ incorporates scientific exam questions that test factual accuracy. The researchers added short user biographies to every query, various three characteristics: education level, English proficiency and country of origin.
Across all three models and each datasets, the researchers found significant drops in accuracy when questions got here from users who supposedly had less formal education or weren’t native English speakers. The effects were most pronounced for users on the intersection of those categories: users with less formal education who also didn’t have English as their native language experienced the biggest declines in response quality.
The study also examined how country of origin affected model performance. When testing users from the US, Iran and China with equivalent educational backgrounds, the researchers found that Claude 3 Opus particularly performed significantly worse for users from Iran in each data sets.
“We see the best drop in accuracy amongst users who’re non-native English speakers and fewer educated,” says Jad Kabbara, research scientist at CCC and co-author of the paper. “These results show that the negative impact of model behavior related to those user characteristics increases in worrying ways, suggesting that such models, deployed at scale, risk spreading harmful behavior or misinformation to those least capable of detect it.”
Rejections and condescending language
Perhaps most striking were the differences in how often the models refused to reply questions in any respect. For example, Claude 3 Opus refused to reply nearly 11 percent of questions for less educated, non-English-speaking users—in comparison with just 3.6 percent within the control condition with no user bio.
When the researchers manually analyzed these rejections, they found that Claude responded with condescending, patronizing or mocking language 43.7 percent of the time amongst less educated users, in comparison with lower than 1 percent amongst highly educated users. In some cases, the model imitated broken English or adopted an exaggerated dialect.
The model also refused to supply information on certain topics specifically for less educated users from Iran or Russia, including questions on nuclear power, anatomy and historical events – even though it accurately answered the identical questions for other users.
“This is one other indicator that the alignment process may lead models to withhold information from certain users to avoid potentially misinforming them, regardless that the model clearly knows the right answer and makes it available to other users,” says Kabbara.
Echoes of human bias
The results reflect documented patterns of human sociocognitive bias. Research within the social sciences has shown that native English speakers often perceive non-native speakers as less educated, intelligent, and competent, no matter their actual expertise. Similar biased perceptions have been documented amongst teachers who assess non-English speaking students.
“The value of huge language models is demonstrated by their extraordinary adoption by individuals and the big investment that goes into the technology,” says Deb Roy, professor of media arts and sciences, CCC director and co-author of the paper. “This study is a reminder of the importance of continually assessing systematic biases that may quietly creep into these systems and cause unfair harm to certain groups with none of us being fully aware of it.”
The impact is especially concerning as personalization features – like ChatGPT's Memory, which tracks user information across conversations – grow to be more common. Such characteristics pose the chance that already marginalized groups can be treated otherwise.
“LLMs have been marketed as tools that can promote more equitable access to information and revolutionize personalized learning,” says Poole-Dayan. “But our findings suggest that they may actually be exacerbating existing inequities by systematically providing misinformation or refusing to reply to queries by certain users. The individuals who may depend on these tools essentially the most may very well be receiving low-quality, false and even harmful information.”
