Mila researchers have proposed a brand new technique that makes large language models (LLMs) significantly more efficient at performing complex reasoning. Called Markovian ponderingThe approach allows LLMs to undertake lengthy reasoning without incurring the prohibitive computational costs that currently limit such tasks.
The team's implementation, an environment called Delethink, structures the reasoning chain into fixed-size blocks, solving the scaling problem that happens with very long LLM answers. Initial estimates show that this method can reduce training costs by greater than two-thirds compared to plain approaches for a 1.5B parameter model.
The square curse of long-chain pondering
For an LLM to unravel a fancy problem, it often must generate a protracted series of intermediate “pondering” tokens, sometimes called a chain-of-think (CoT). In recent years, researchers have discovered that that is the case Reinforcement learning (RL) for training models to create longer CoTs (sometimes known as LongCoT) has significantly improved their reasoning skills.
However, the usual approach to doing this has a critical flaw: the AI's “state” (the prompt plus all of the reasoning tokens it has generated to date in its processing) grows with each recent reasoning token. For modern transformer based modelsThis implies that the computational cost explodes quadratically because the reasoning chain gets longer, making it prohibitively expensive to coach models for very complex tasks.
Most current attempts to administer these costs give attention to limiting the model's amount of pondering, implicitly favoring shorter solutions, or prematurely terminating the method. Although these methods provide some relief, Mila researchers are still working inside the LongCoT framework and are subsequently fundamentally certain by its quadratic nature.
Instead of trying to manage computational growth, Mila created an RL environment that completely avoids the quadratic problem. As co-author Amirhossein Kazemnejad explained, the goal is to enable skills reminiscent of multi-week pondering and scientific discovery. “This regime (and the RL required to enable such capabilities) is just not supported by the present LongCoT paradigm because of the squared computational cost,” he said.
Think in blocks with Delethink
The researchers' solution is a paradigm they call the “Markovian Thinker,” wherein the model argues while the dimensions of its argument context window stays constant. The core idea is to vary the RL setup to separate “how long the model thinks” and “how much context it must process”. When done accurately, a Markovian Thinker converts the quadratic growth problem into linear computing power and glued storage requirements for LLM conclusions.
The researchers put this paradigm into practice through Delethink, which forces the model to reason in a sequence of blocks of a set size, for instance 8,000 tokens at a time. Within each block, the model argues as usual, using the classic attention mechanism. However, when the limit of the block is reached, the environment resets the context and creates a brand new prompt containing the unique query in addition to a brief “transmission” from the previous block. For example, the carryover may very well be the previous few tokens of the previous CoT part or a summary of the important thing results.
This reframing of the issue forces the model to learn how you can embed a summary of its progress or a “textual Markovian state” into this carry as a way to proceed its reasoning in the subsequent section. This addresses the common concern about whether the model can remember essential details from previous steps.
According to Kazemnejad, the model learns what it needs to recollect. “Through training… the model is forced to learn to proceed the task-critical state,” he explained. He added a vital practical clarification: the unique prompt is just not modified, including any documents or contextual data added to it. “Our approach targets the argument phase and doesn’t change the prompt,” he said.
Delethink in motion
To test their approach, the researchers trained R1-Distill-1.5B with Delethink on a dataset of competitive-level math problems after which evaluated it against several benchmarks. The model was trained to process as much as 24,000 tokens, but with fixed 8,000 token blocks.
The researchers compared this to models trained using the usual LongCoT-RL method. Their results indicate that the model trained with Delethink was in a position to handle as much as 24,000 tokens and on mathematical benchmarks matched or outperformed a LongCoT model trained on the identical budget of 24,000 tokens. In other tasks reminiscent of coding and doctoral-level questions, Delethink was also in a position to match or barely outperform its LongCoT counterpart. “Overall, these results suggest that Delethink uses its pondering tokens just as effectively as LongCoT-RL with reduced computing power,” the researchers write.
When scaling beyond the training budget, the advantages develop into much more apparent. While models trained with LongCoT quickly reached their training limits, the model trained with Delethink continued to enhance its performance. For example, some math problems were solved only after the model considered as much as 140,000 tokens, well beyond its training budget of 24,000 tokens. This linear computing advantage is important for enterprise applications. The researchers estimate that training a model to a median think length of 96,000 tokens with LongCoT would require 27 H100 GPU months, versus just 7 with Delethink.
This efficiency extends on to inference, which is the first operating cost for many organizations. “Models trained in Markovian pondering use the identical inference style (delethink tracing) during test time, which provides the identical advantages of linear computation and constant memory after training,” Kazemnejad said. He provided a practical example: an AI agent could “debug a big code base and think for a very long time…which in fact significantly reduces the associated fee in comparison with the standard LongCoT approach.”
Interestingly, the researchers found that commercially available argumentation models already show a certain ability for Markovian pondering, even without special training. This finding has immediate practical implications for developers. “In practice, because of this these models – without Delethink-RL – already run a Delethink tracing wrapper and will be competitive with LongCoT on our benchmark tasks,” Kazemnejad said.
Your experiments with larger models reminiscent of GPT-OSS 120B demonstrated robust performance across a spread of complex tasks using Delethink. This latent ability provides a robust place to begin for RL training and helps explain why the tactic is so effective. “Taken together, these results suggest that Delethink is compatible with state-of-the-art models and scales,” the researchers conclude.
The success of Markovian pondering shows that it could be possible for “next-generation pondering models to think for tens of millions of tokens,” the researchers note. This opens the door to fundamentally recent AI capabilities that transcend current limitations.
“Markovian pondering… opens the solution to models that may 'think' over very long horizons, which we see as a essential step toward eventual scientific discovery,” Kazemnejad said. “Our approach removes a crucial bottleneck and may enable training for tasks with for much longer horizons, enabling next-generation capabilities.”
