Applied AI research group Nous Research has developed a training optimizer for AI models that would dramatically change the best way AI models are trained in the long run.
Traditionally, training an AI model requires huge data centers filled with GPUs like NVIDIA's H100s, in addition to high-speed interconnects to synchronize gradient and parameter updates between GPUs.
Each training step requires exchanging huge amounts of information between 1000’s of GPUs. The bandwidth required implies that these GPUs have to be hardwired and physically close to one another. With DisTrO, Nous Research can have found a approach to change this completely.
As a model is trained, an optimization algorithm adjusts the model's parameters to attenuate the loss function. The loss function measures the difference between the model's predictions and the actual results. The goal is to cut back this loss as much as possible through iterative training.
DisTrO-AdamW is a variant of the favored optimization algorithm AdamW. DisTrO stands for “Distributed Training Over-the-Internet” and indicates what makes it so special.
DisTrO-AdamW drastically reduces the quantity of inter-GPU communication required during training of huge neural networks without compromising the convergence rate or accuracy of the training process.
In empirical tests, DisTrO-AdamW achieved an 857-fold reduction in inter-GPU communication. This implies that the DisTrO approach can train models with comparable accuracy and speed without the necessity for expensive high-bandwidth hardware.
For example, when pretraining a 1.2 billion LLM, DisTrO-AdamW achieved the identical performance as conventional methods while reducing the required bandwidth from 74.4 GB to only 86.8 MB per training step.
What in case you could harness all of the computing power on this planet to coach a standard open-source AI model?
Preliminary report: https://t.co/b1XgJylsnV
Nous Research is proud to publish a preliminary report on DisTrO (Distributed Training Over-the-Internet), a family of… pic.twitter.com/h2gQJ4m7lB
— Nous Research (@NousResearch) 26 August 2024
Impact on AI training
The impact of DisTrO on the AI landscape could possibly be profound. By reducing communication overhead, DisTrO enables decentralized training of huge models. Instead of an information center with 1000’s of GPUs and high-speed switches, you would train a model on distributed industrial hardware connected over the web.
You could have a community of individuals providing access to their computer hardware to coach a model. Imagine thousands and thousands of unused PCs or redundant Bitcoin mining rigs working together to coach an open source model. DisTrO makes this possible, and there’s little penalty within the time required to coach the model or its accuracy.
Nous Research admits that they usually are not really sure why their approach works so well and that further research is required to see whether it is transferable to larger models.
If successful, it’ll mean that training huge models will now not be dominated by the massive tech corporations which have the cash to construct huge data centers. It could even have a big effect by reducing the environmental impact of energy- and water-hungry data centers.
The concept of decentralized training could also render obsolete some elements of regulations comparable to California's proposed bill, SB 1047. The bill would require additional safety checks for models that cost greater than $100 million to coach.
With DisTrO, a community of anonymous people could use distributed hardware to create their very own “supercomputer” to coach a model. It could also US Government efforts to stop China from importing NVIDIA's strongest GPUs.
In a world where AI is becoming increasingly essential, DisTrO offers a glimpse right into a future where the event of those powerful tools is more comprehensive, sustainable and widespread.
