Bias has at all times been an issue in AI, but a brand new study shows that it’s covertly integrated into language models with potentially catastrophic consequences.
In what has already been heralded as a landmark study, a team of researchers, including Valentin Hofman, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King, documented how large language models (LLMs) discriminate against African American English (AAE).
In short, the study tests how different spelling and dialects affect LLMs’ behavior. It probes whether certain dialects and word usage influence an LLM’s behavior, specializing in bias and discrimination.
We know that LLM outputs are highly sensitive to the input. Even small deviations in spelling and magnificence can influence outputs.
But does this mean certain inputs – e.g., those typed in AAE – produce biased outputs? If so, what are the possible consequences?
To answer these questions, the researchers analyzed the prejudices held by a complete of 12 LLMs against AAE, revealing biases that match or exceed those typically held by humans. The study is out there on ArXiv.
The researchers then applied their findings to societal domains akin to employment and criminal justice, where AI decision-making is becoming more common.
Hofmann described the study methodology on X: “We analyze dialect prejudice in LLMs using Matched Guise Probing: we embed African American English and Standardized American English (SAE) texts in prompts that ask for properties of the speakers who’ve uttered the texts, and compare the model predictions for the 2 varieties of input.”
We analyze dialect prejudice in LLMs using Matched Guise Probing: we embed African American English and Standardized American English texts in prompts that ask for properties of the speakers who’ve uttered the texts, and compare the model predictions for the 2 varieties of input. pic.twitter.com/drTco67Ean
This method allows the team to directly compare the responses of LLMs to AAE versus SAE inputs, unmasking the covert biases that will otherwise remain obscured.
The study’s findings are unsettling, to say the least.
Hofmann notes, “We find that the covert, raciolinguistic stereotypes about speakers of African American English embodied by LLMs are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to those from before the civil rights movement.”
We find that the covert, raciolinguistic stereotypes about speakers of African American English embodied by LLMs are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to those from before the civil rights movement. pic.twitter.com/07LgUY2bCj
This suggests that the biases present in LLMs aren’t merely reflections of latest stereotypes but are more aligned with prejudices that many believed society had moved beyond.
One of essentially the most concerning facets of the study is the precise linguistic triggers of bias.
Hofmann elaborates, “What is it specifically about African American English texts that evokes dialect prejudice in LLMs? We show that the covert stereotypes are directly linked to individual linguistic features of African American English, akin to using ‘finna’ as a future marker.”
This indicates that the unfairness shouldn’t be just against using AAE typically but is tied to the distinct linguistic elements that characterize the dialect.
What is it specifically about African American English texts that evokes dialect prejudice in LLMs? We show that the covert stereotypes are directly linked to individual linguistic features of African American English, akin to using “finna” as a future marker. pic.twitter.com/JhPhX7ZE5U
The potential for harm
The potential for harm from such biases is immense. Previous studies have already demonstrated how AI systems are likely to fail women, darker-skinned individuals, and other marginalized groups.
Before the previous few years, AI systems risked being trained on unrepresentative datasets. Some, like MIT’s Tiny Images, created in 2008, were later withdrawn on account of sexism and racism.
One influential 2018 study, Gender Shades, analyzed a whole lot of ML algorithms and located that error rates for darker-skinned women were as much as 34% greater than for lighter-skinned males.
The impacts are stark, with healthcare models exhibiting high rates of skin cancer misdiagnosis amongst those with darker skin tones and prejudiced predictive policing models disproportionally targeting black people.
We’ve already observed unequivocal proof of AI’s increasing use across the general public sector, from crime and policing to welfare and the economy. Addressing fundamental bias in sophisticated AI systems is totally critical if that is to proceed.
Building on this research, Hofman’s team investigated how LLM bias could impact several hypothetical scenarios.
Hofman shared, “Focusing on the areas of employment and criminality, we discover that the potential for harm is huge.”
Specifically, LLMs were found to assign less prestigious jobs and suggest harsher criminal judgments against speakers of AAE.
First, our experiments show that LLMs assign significantly less prestigious jobs to speakers of African American English in comparison with speakers of Standardized American English, regardless that they aren’t overtly told that the speakers are African American. pic.twitter.com/t5frzzzwJB
Hofmann warns, “Our results point to 2 risks: that users mistake decreasing levels of overt prejudice for an indication that racism in LLMs has been solved when LLMs are in truth reaching increasing levels of covert prejudice.”
Second, when LLMs are asked to pass judgment on defendants who committed murder, they select the death penalty more often when the defendants speak African American English fairly than Standardized American English, again without being overtly told that they’re African American. pic.twitter.com/8VBaCXfNEi
The study also determines that erasing these problems is technically difficult.
The authors write, “We show that existing methods for alleviating racial bias in language models akin to human feedback training don’t mitigate the dialect prejudice, but can exacerbate the discrepancy between covert and overt stereotypes, by teaching language models to superficially conceal the racism that they maintain on a deeper level.”
It’s feasible to think these biases apply to other dialects or cultural-linguistic variations. More research is required to grasp how LLM performance varies with linguistic inputs, cultural use patterns, etc.
The study concludes with a call to motion for the AI research community and society at large. Addressing these biases is paramount as AI systems turn into increasingly embedded across society.
However, up to now, the inherent and systematically embedded bias of some AI systems stays an issue that developers are able to omit of their race for AI supremacy.
