Large language models (LLMs) are excellent at answering easy questions, but require special prompting techniques to handle complex tasks that require logical pondering and planning. These prompting schemes, sometimes called “System 2” techniques, improve the reasoning skills of LLMs by forcing them to generate intermediate steps to resolve an issue.
System 2 techniques, while effective, make LLM applications slow and computationally intensive. In a brand new paper, researchers from MetaFAIR currently “System 2 Distillation”, a way that teaches LLMs complex tasks without intermediate steps.
System 1 and System 2 in Cognitive Science and LLMs
In cognitive science, System 1 and System 2 check with two alternative ways of pondering. System 1 pondering is fast, intuitive, and automatic. We use it once we recognize patterns, make quick judgments, or understand familiar symbols. For example, we use System 1 pondering to discover traffic signs, recognize faces, and associate easy symbols with their meanings.
System 2 pondering, then again, is slow, deliberate, and analytical. It requires conscious effort and is used for complex problem solving, resembling manipulating abstract symbols, solving mathematical equations, or planning a visit.
LL.M. graduates are generally considered to be analogous to System 1 pondering. They can generate text in a short time, but have difficulty with tasks that require deliberate pondering and planning.
In recent years, AI researchers have shown that LLMs might be made to mimic System 2 pondering by asking them to generate intermediate steps in reasoning before giving their final answer. For example, “Chain of Thought” is a prompting technique that instructs the LLM to elucidate its thought process step-by-step, which regularly results in more accurate leads to logical reasoning tasks. Several System 2 prompting techniques are tailored to different tasks.
“Many of those methods have been shown to provide more accurate results as a result of this explicit reasoning, but typically accomplish that at much higher inference costs and latency to a response,” write the Meta AI researchers. “For the latter reason, a lot of these approaches should not utilized in production systems, which mostly use System 1 generations.”
System 2 Distillation
An interesting commentary about System 2 pondering in humans is that when a task requires conscious effort, it regularly becomes ingrained in our System 1 when performed repeatedly. For example, whenever you learn to drive a automotive, you spend numerous conscious effort steering the automotive, obeying traffic rules, and navigating. But as you gain experience, driving becomes second nature. You not must take into consideration each step and might perform it intuitively and mechanically.
This phenomenon inspired Meta-AI researchers to develop “System 2 distillation” for LLMs.
Distillation is a standard technique in machine learning (ML) where a bigger model, called the “teacher,” is used to coach a smaller model, called the “student.” For example, developers often use pioneer models resembling GPT-4 and Claude to generate training examples for smaller models resembling Llama-2 7B.
However, System 2 distillation doesn’t use a separate teacher model. Instead, researchers found a approach to distill the knowledge gained from the model's System 2 reasoning skills into fast and computationally efficient System 1 generation.
The process begins by asking the LLM to resolve an issue using System 2 input techniques. The answers are then checked for correctness by an unsupervised mechanism. For example, “self-consistency” is used, where the model is given the identical input multiple times. The answers are then compared and probably the most often occurring answer is taken into account the proper answer and chosen for the distillation dataset. If the answers are too inconsistent, the instance and its answers are discarded.
Next, they discard the intermediate steps generated by the System 2 reasoning process and keep only the ultimate answers. Finally, they refine the model based on the unique query and the reply. This allows the model to skip the reasoning steps and go on to the reply.
System 2 distillation in motion
The researchers evaluated their method using a series of pondering tasks and 4 different System 2 input techniques. As a baseline model, they used Llama-2-70B, which is large enough to internalize latest knowledge.
The System 2 approaches they utilized in their experiments include Chain-of-Thought, System 2 Attention, Rephrase and respond and Branch-Solve-Merge. Some of those techniques require the model to be asked multiple times, making them each slow and expensive. For example, Rephrase and Respond first asks the model to reformulate the unique query with elaboration, after which asks the model again with the reformulated query. Branch-Solve-Merge is much more complicated and requires multiple back-and-forth operations with the model.
The results show that System 2 distillation can significantly improve the performance of LLMs on complex reasoning tasks, often meeting or exceeding the accuracy of the unique System 2 methods. In addition, the distilled models can generate answers much faster and with less computational effort because they do not need to undergo the intermediate steps of the reasoning process.
For example, they found that distillation was successful in tasks that used System 2 Attention to cope with biased opinions or irrelevant information. It also showed impressive leads to some reasoning tasks that used Rephrase and Respond to elucidate and improve answers, in addition to for fine-grained evaluation and processing of tasks by Branch-Unlink-Merge.
“We have shown that in lots of cases it is feasible to distill this System 2 reasoning into the outputs of the LLM without intermediate generations while maintaining or sometimes even improving performance,” the researchers write.
However, the researchers also found that LLMs, like humans, cannot integrate every type of pondering skills into their rapid reasoning mechanism. For example, they were unable to successfully integrate complex mathematical reasoning tasks that Thought chain suggestionThis suggests that some tasks at all times require careful pondering.
There is way more to find out about System 2 distillation, resembling how well it performs on smaller models and the way distillation affects the model's overall performance on tasks that weren’t included within the distillation training dataset. It can be price noting that LLM benchmarks are sometimes susceptible to contamination when the model already has some knowledge of the test examples, resulting in inflated results on test sets.
However, for mature LLM pipelines that perform specific tasks at each step, distillation is definitely a strong optimization tool.
“In the long run, systems that may filter out useful tasks in this fashion could have more time to think concerning the tasks they can’t yet do well – similar to humans,” the researchers write.
