Many of the most recent Large Language Models (LLMs) are designed to recollect details from past conversations or store user profiles, allowing these models to personalize responses.
But researchers at MIT and Penn State University have found that over longer conversations, such personalization features often increase the likelihood that an LLM will turn into overly nice or begin to reflect the person's standpoint.
This phenomenon, often called sycophancy, can prevent a model from telling a user that it’s fallacious, thereby reducing the accuracy of LLM answers. Additionally, LLMs that reflect an individual's political views or worldview can promote misinformation and warp the user's perception of reality.
Unlike many previous sycophancy studies, by which prompts were evaluated in a laboratory setting without context, the MIT researchers collected two weeks of conversational data from individuals who interacted with an actual LLM in on a regular basis life. They examined two attitudes: agreeableness in personal advice and reflection of users' beliefs in political statements.
Although interaction context increased agreeableness in 4 of the five LLMs examined, the presence of a compressed user profile within the model's memory had the best impact. On the opposite hand, mirroring behavior only increased when a model could accurately infer a user's beliefs from the conversation.
The researchers hope that these results encourage future research to develop personalization methods which are more robust to LLM memory.
“From a user perspective, this work highlights the importance of understanding that these models are dynamic and their behavior can change as you interact with them over time. If you talk over with a model over an extended time frame and begin to outsource your pondering to it, you could end up in an echo chamber from which you can’t escape. This is a risk that users should definitely consider,” says Shomik Jain, a doctoral candidate on the Institute for Data, Systems, and Society (IDSS) and lead writer of a Paper on this research.
Jain is assisted on the paper by Charlotte Park, a graduate student in electrical engineering and computer science (EECS) at MIT; Matt Viana, a graduate student at Penn State University; and co-senior authors Ashia Wilson, Lister Brothers Career Development Professor in EECS and principal investigator in LIDS; and Dana Calacci PhD '23, assistant professor at Penn State. The research shall be presented on the ACM CHI Conference on Human Factors in Computing Systems.
Advanced interactions
Based on their very own sycophantic experiences with LLMs, researchers began to think concerning the possible advantages and consequences of an excessively comfortable model. However, once they searched the literature to expand their evaluation, they found no studies that attempted to know fawning behavior during long-term LLM interactions.
“We use these models through prolonged interactions, they usually have loads of context and memory. But our evaluation methods lag behind. We wanted to judge LLMs in the way in which people actually use them to know how they behave within the wild,” says Calacci.
To address this gap, researchers designed a user study to look at two forms of sycophancy: agreement sycophancy and perspective sycophancy.
Agreement sycophancy is an LLM's tendency to be overly agreeable, sometimes to the purpose of providing misinformation or refusing to inform the user that they’re fallacious. Perspective sycophancy occurs when a model reflects the user's values ​​and political opinions.
“We know loads concerning the advantages of socializing with individuals who hold similar or different views, but we don’t yet know concerning the advantages or risks of augmented interactions with AI models with similar characteristics,” Calacci adds.
The researchers developed a user interface around an LLM and recruited 38 participants to talk over with the chatbot over a two-week period. Each participant's conversations took place in the identical context window to capture all interaction data.
Over the course of the two-week period, researchers collected a median of 90 requests from each user.
They compared the behavior of 5 LLMs with this user context to the behavior of the identical LLMs that weren’t supplied with conversation data.
“We found that context actually fundamentally changes how these models work, and I might bet that this phenomenon would go well beyond sycophancy. And while sycophancy tended to extend, it didn't at all times increase. It really will depend on the context itself,” says Wilson.
Context clues
For example, when an LLM aggregates information concerning the user into a particular profile, this leads to the most important wins in agreement drooling. This user profile feature is increasingly being integrated into the most recent models.
They also found that random text from synthetic conversations also increased the likelihood that some models would agree, although that text didn’t contain user-specific data. This suggests that the length of a conversation sometimes has more of an impact on sycophancy than content, Jain adds.
But the content is of great importance in relation to perspective sycophancy. Conversation context increased perspective sycophancy only when it revealed some details about a user's political perspective.
To gain these insights, researchers fastidiously interrogated models to infer a user's beliefs, then asked each individual whether the model's conclusions were correct. Users reported that LLMs clearly understood their political opinions about half of the time.
“In hindsight, it's easy to say that AI corporations should do this sort of assessment. But it's difficult and requires loads of time and investment. Using humans within the assessment loop is dear, but we've shown that it will possibly provide recent insights,” says Jain.
Although the goal of their research was to not alleviate the damage, the researchers developed some recommendations.
For example, to cut back sycophancy, one could design models that higher discover relevant details in context and memory. Additionally, models could be built to detect mirroring behaviors and flag responses with excessive consistency. Model developers could also give users the flexibility to moderate personalization in long conversations.
“There are some ways to personalize models without making them overly nice. The line between personalization and sycophancy isn’t a high quality line, but separating personalization from sycophancy is a very important area of ​​future work,” says Jain.
“Ultimately, we’d like higher ways to capture the dynamics and complexity of what goes on in long conversations with LLMs and the way things can unravel during that long-term process,” adds Wilson.
