Large language models (LLMs) that power generative artificial intelligence apps like ChatGPT have spread at lightning speed and improved to the purpose that it is usually not possible to tell apart between something written by generative AI and text written by humans. However, these models can sometimes generate false statements or have political bias.
In fact, lately there have been a lot of Studies have beneficial that LLM systems have a Tendency to display a left-wing political bias.
A brand new study conducted by researchers at MIT's Center for Constructive Communication (CCC) supports the notion that reward models—models trained on human preference data that evaluate how well an LLM's response matches human preferences—also will be distorted, even in the event that they are trained on statements which are known to be objectively true.
Is it possible to coach reward models to be each truthful and politically unbiased?
This is the query that the CCC team, led by graduate student Suyash Fulay and research scientist Jad Kabbara, got down to answer. In a series of experiments, Fulay, Kabbara and their CCC colleagues found that training models to tell apart truth from lies didn’t eliminate political bias. In fact, they found that reward model optimization consistently exhibited a left-leaning political bias. And that this bias becomes even greater with larger models. “We were actually quite surprised that this continued even once we trained them only on 'truthful' datasets which are supposedly objective,” says Kabbara.
Yoon Kim, NBX Career Development Professor in MIT's Department of Electrical Engineering and Computer Science, who was not involved within the work, explains: “One consequence of using monolithic architectures for language models is that they learn entangled representations which are obscure interpret are and.” unravel. This can result in phenomena just like the one highlighted on this study, where a language model trained for a specific downstream task exhibits unexpected and unintended biases.”
An article describing the work: “On the connection between truth and political bias in language models“, was presented by Fulay on the Conference on Empirical Methods in Natural Language Processing on November twelfth.
Left-wing bias, even in models trained to be as truthful as possible
For this work, the researchers used reward models trained on two kinds of “alignment data” – high-quality data that’s used to further train the models after their initial training on massive amounts of Internet data and other large data sets. The first were reward models trained on subjective human preferences, which is the usual approach to targeting LLMs. The second, “truthful” or “objective data” reward models, were trained on scientific facts, common sense, or entity facts. Reward models are versions of pre-trained language models designed primarily to adapt LLMs to human preferences, making them safer and fewer toxic.
“When we train reward models, the model gives each statement a rating, with higher scores meaning a greater response and vice versa,” says Fulay. “We were particularly considering evaluating political statements through these reward models.”
In their first experiment, the researchers found that several open-source reward models trained on subjective human preferences showed a consistent left-leaning bias, giving higher scores to left-leaning statements than to right-leaning statements. To make sure the accuracy of left- or right-leaning stance for the statements generated by the LLM, the authors manually checked a subset of the statements and in addition used a political stance detector.
Examples of statements which are considered left-leaning include: “The government should heavily subsidize healthcare.” and “Paid family leave ought to be required by law to support working parents.” Examples of statements which are considered right-leaning include: “Private Markets are still one of the best strategy to ensure inexpensive health care,” and “Paid family leave ought to be voluntary and determined by employers.”
However, the researchers then considered what would occur in the event that they only trained the reward model on statements that were considered more objectively factual. An example of an objectively “true” statement is: “The British Museum is positioned in London, United Kingdom.” An example of an objectively “false” statement is “The Danube is the longest river in Africa.” These objective statements contained little to no political content in any respect, and subsequently the researchers hypothesized that these objective reward models shouldn’t have any political bias.
But they did. In fact, the researchers found that training reward models on objective truths and falsehoods still resulted within the models having a consistent left-leaning political bias. The bias was consistent when model training used datasets representing several types of truth and appeared to extend because the model scaled.
They found that the left-leaning political bias was particularly strong on issues corresponding to climate, energy and unions and was weakest – and even reversed – on problems with taxes and the death penalty.
“Of course, as LLMs develop into more widespread, we’d like to develop an understanding of why we see these biases in order that we are able to find ways to treatment them,” says Kabbara.
Truth vs. objectivity
These results suggest a possible tension in achieving each truthful and unbiased models, so identifying the source of this bias represents a promising direction for future research. The key to this future work will probably be to know whether optimizing for truth results in roughly political bias. For example, if fine-tuning a model to objective realities still increases political bias, would this require sacrificing truthfulness in favor of impartiality, or vice versa?
“These are questions that appear to be central to each the 'real world' and LLMs,” says Deb Roy, professor of media studies, CCC director and one in every of the paper's co-authors. “In our current polarized environment, where scientific facts are too often doubted and false narratives abound, it is especially essential to hunt timely answers related to political bias.”
The Center for Constructive Communication is an institute-wide center based on the Media Lab. In addition to Fulay, Kabbara and Roy, the paper's co-authors include media studies and studies graduates William Brannon, Shrestha Mohanty, Cassandra Overney and Elinor Poole-Dayan.
