Large language models, resembling those powered by popular artificial intelligence chatbots like ChatGPT, are incredibly complex. Although these models are used as tools in lots of areas, resembling customer support, code generation, and language translation, scientists still don’t fully understand how they work.
To higher understand what's happening under the hood, researchers at MIT and elsewhere examined the mechanisms at work when these massive machine learning models retrieve stored knowledge.
They found a surprising result: large language models (LLMs) often use a quite simple linear function to recuperate and decode stored facts. In addition, the model uses the identical decoding function for similar varieties of facts. Linear functions, equations with only two variables and no exponents, capture the direct, straight-line relationship between two variables.
The researchers showed that by identifying linear functions for various facts, they’ll examine the model to see what it knows about latest topics and where throughout the model that knowledge is stored.
Using a way they developed to estimate these easy functions, the researchers found that even when a model answered an issue incorrectly, it often retained the proper information. In the long run, scientists could use such an approach to seek out and proper falsehoods throughout the model, which could reduce a model's tendency to sometimes give incorrect or nonsensical answers.
“Although these models are really complicated, nonlinear functions which are trained on a variety of data and are very obscure, sometimes really easy mechanisms work in them. This is an example of that,” says Evan Hernandez, an electrical engineering and computer science (EECS) graduate student and co-lead writer of a Paper detailing these results.
Hernandez wrote the paper with co-lead writer Arnab Sharma, a pc science doctoral student at Northeastern University; his advisor Jacob Andreas, associate professor of EECS and member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); lead writer David Bau, assistant professor of computer science at Northeastern; and others at MIT, Harvard University and the Israeli Institute of Technology. The research will likely be presented on the International Conference on Learning Representations.
Find facts
Most large language models, also called transformer models, are neural networks. Neural networks are loosely based on the human brain and contain billions of interconnected nodes, or neurons, grouped into many layers that encode and process data.
Much of the knowledge stored in a transformer may be represented as relationships connecting subjects and objects. For example, “Miles Davis plays the trumpet” is a relationship that connects the topic Miles Davis to the thing trumpet.
As a Transformer gains more knowledge, it stores additional facts about a specific topic across multiple levels. When a user asks about this topic, the model must decode essentially the most relevant fact to answer the query.
If someone requests a transformer by saying, “Miles Davis plays that. . .” The model should answer “trumpet,” not “Illinois” (the state where Miles Davis was born).
“Somewhere within the network computation there should be a mechanism that appears for the incontrovertible fact that Miles Davis is playing the trumpet after which pulls out that information and helps generate the following word. We wanted to grasp what this mechanism was,” says Hernandez.
The researchers conducted a series of experiments to review LLMs and located that, although the models are extremely complex, they decode relational information using an easy linear function. Each function is restricted to the sort of fact retrieved.
For example, the transformer would use a decoding function each time it desired to output the instrument an individual plays, and a special function each time it desired to output the state wherein an individual was born.
The researchers developed a technique for estimating these easy functions after which calculated functions for 47 different relationships, resembling “capital of a rustic” and “lead singer of a band.”
Although there could possibly be an infinite variety of possible relationships, the researchers selected to review this specific subset because they’re representative of the varieties of facts that may be written this manner.
They tested each feature by changing the topic to see if it could restore the proper object information. For example, the function for Capital of a Country should retrieve Oslo if the topic is Norway and London if the topic is England.
Functions retrieved the proper information greater than 60 percent of the time, showing that some information in a transformer is encoded and retrieved this manner.
“But not the whole lot is linearly coded. For some facts, we cannot find linear functions for them, regardless that the model knows them and predicts text that matches those facts. This suggests that the model is doing something more complicated to store this information,” he says.
Visualizing the knowledge of a model
They also used the features to find out what a model believes to be true on various topics.
In one experiment, they began with the prompt “Bill Bradley was a” and used the decoding functions for “plays sports” and “attended college” to see if the model knows that Senator Bradley was a basketball player who attended Princeton.
“We can show that even when the model chooses to give attention to other information when producing text, it still encodes all of that information,” says Hernandez.
They used this probing technique to create what they call an “attribute lens,” a grid that visualizes where specific details about a specific relationship is stored within the transformer's many layers.
Attribute lenses may be mechanically generated and supply a streamlined method for researchers to learn more a few model. This visualization tool could allow scientists and engineers to correct stored knowledge and stop an AI chatbot from providing misinformation.
In the long run, Hernandez and his colleagues want to raised understand what happens when facts usually are not stored linearly. You also wish to conduct experiments with larger models and study the precision of linear decoding functions.
“This is exciting work that uncovers a missing a part of our understanding of how large language models retrieve factual knowledge during inference.” Previous work has shown that LLMs create information-rich representations of particular subjects from which specific attributes are extracted during inference. “This work shows that the complex nonlinear computation of LLMs for attribute extraction may be well approximated with an easy linear function,” says Mor Geva Pipek, assistant professor on the School of Computer Science at Tel Aviv University, who was not involved on this work.
This research was supported partly by Open Philanthropy, the Israeli Science Foundation, and an Early Career Faculty Fellowship from the Azrieli Foundation.
