Large language models can do impressive things, like write poetry or create usable computer programs, although these models are trained to predict words that may come next in a paragraph of text.
Such surprising abilities can provide the impression that the models are implicitly learning some general truths concerning the world.
However, in accordance with a brand new study, that's not necessarily the case. The researchers found that a preferred form of generative AI model can provide detailed directions in New York City with near-perfect accuracy – without creating an accurate internal map of the town.
Despite the model's amazing ability to navigate effectively, its performance plummeted when researchers closed some roads and added detours.
As they dug deeper, the researchers found that the New York City maps that the model implicitly generated contained many non-existent streets that ran between the grid and distant intersections.
This could have serious implications for generative AI models deployed in the actual world, as a model that appears to work well in a given context may break down if the duty or environment changes barely.
“One hope is that because LLMs can do all these amazing things in language, perhaps we could use these same tools in other areas of science.” But the query of whether LLMs learn coherent models of the world could be very necessary if “We wish to use these techniques to make latest discoveries,” says lead creator Ashesh Rambachan, assistant professor of economics and principal investigator on the MIT Laboratory for Information and Decision Systems (DECKEL).
Rambachan is connected to a Paper about work by lead creator Keyon Vafa, a postdoctoral fellow at Harvard University; Justin Y. Chen, an electrical engineering and computer science (EECS) graduate student at MIT; Jon Kleinberg, Tisch University Professor of Computer and Information Science at Cornell University; and Sendhil Mullainathan, MIT professor within the EECS and economics departments and member of LIDS. The research shall be presented on the Conference on Neural Information Processing Systems.
New metrics
The researchers focused on a form of generative AI model often called Transformer, which forms the backbone of LLMs like GPT-4. Transformers are trained on a large amount of language-based data to predict the following token in a sequence, comparable to the following word in a sentence.
But if scientists want to find out whether an LLM has created an accurate model of the world, measuring the accuracy of its predictions isn’t enough, the researchers say.
For example, they found that a transformer can predict valid moves almost each time in a game of Connect 4 without understanding the foundations.
So the team developed two latest metrics that might be used to check a transformer's world model. The researchers focused their evaluations on a category of problems called deterministic finite automations, or DFAs.
A DFA is an issue with a sequence of states, comparable to intersections that one must cross to achieve a destination, and a concrete description of the foundations one must follow along the way in which.
They selected two problems to formulate as DFAs: navigating the streets of New York City and playing the board game Othello.
“We needed test benches where we knew the world model. Now we will think deeply about what it means to revive this world model,” explains Vafa.
The first metric they developed, called sequence discrimination, says that a model has formed a coherent model of the world when it sees two different states, like two different Othello boards, and recognizes how different they’re. Sequences, that are ordered lists of information points, are utilized by transformers to generate outputs.
The second metric, called sequence compression, states that a transformer with a coherent world model should know that two an identical states, like two an identical Othello boards, have the identical order of possible next steps.
They used these metrics to check two common classes of transformers, one trained on data generated from randomly generated sequences and the opposite on data generated by following strategies.
Incoherent world models
Surprisingly, the researchers found that Transformers who made decisions randomly formed more accurate models of the world, perhaps because they saw a greater number of possible next steps during training.
“In Othello, in case you see two computers playing at random as a substitute of championship players, you theoretically see all possible moves, even the bad moves that championship players wouldn't make,” explains Vafa.
Although the transformers generated accurate directions and valid Othello movements in almost every case, the 2 metrics showed that just one generated a coherent world model for Othello movements and neither performed well in forming coherent world models within the pathfinding example.
The researchers demonstrated the consequences by inserting detours into the map of New York City, which caused all navigation models to fail.
“I used to be surprised at how quickly performance dropped as soon as we added a detour. If we close just 1 percent of possible roads, the accuracy immediately drops from almost one hundred pc to simply 67 percent,” says Vafa.
When they recreated the town maps generated by the models, they looked like an imaginary New York City with lots of of streets criss-crossed overlaid on the grid. The maps often contained random overpasses over other streets or multiple streets with unattainable alignments.
These results show that transformers can perform surprisingly well on certain tasks without understanding the foundations. If scientists wish to construct LLMs that may capture accurate models of the world, they should take a distinct approach, the researchers say.
“We often see these models doing impressive things and think they will need to have understood something concerning the world. “I hope we will persuade those that this can be a query that should be thought of very rigorously and that we don’t need to depend on our own intuition to reply it,” says Rambachan.
In the long run, researchers wish to tackle more diverse problems, comparable to those where some rules are only partially known. They also wish to apply their evaluation standards to real, scientific problems.
This work is funded partially by the Harvard Data Science Initiative, a National Science Foundation Graduate Research Fellowship, a Vannevar Bush Faculty Fellowship, a Simons Collaboration Grant, and a MacArthur Foundation grant.
