In an exclusive interview with VentureBeat, Itamar ArelFounder and CEO of an AI startup Tenyxrevealed a groundbreaking achievement in the sector of natural language processing. Tenyx has successfully refined Meta's (now generally known as) Llama-3 open source language model Tenyx-70B) to surpass OpenAI's GPT-4 in certain areas, marking the primary time an open source model has surpassed the proprietary gold standard.
“We developed this fine-tuning technology that enables us to take a basic model and kind of refine it or train it beyond what it was trained on,” Arel explained. “What we're getting an increasing number of enthusiastic about is that we could use this technology, which essentially allows us to take advantage of some redundancies in these large models, to enable what would probably be higher called continuous learning or incremental learning. “
Overcoming “catastrophic forgetting”.
Tenyx's novel fine-tuning approach addresses the issue of “catastrophic forgetting,” where a model loses previously learned knowledge when exposed to latest data. By selectively updating only a small portion of the model parameters, Tenyx can efficiently train the model on latest information without compromising its existing capabilities.
“If you find yourself changing, say, just 5% of the model parameters and the whole lot else stays the identical, you may be more aggressive without the danger of distorting other things,” Arel said. This selective parameter updating method has also enabled Tenyx to attain remarkably fast training times, refining the 70 billion parameter Llama-3 model in only 15 hours using 100 GPUs.
Commitment to open source AI
Tenyx's commitment to open source AI is reflected in the choice to release its fine-tuned model named Tenyx-70B, under the identical license as the unique Llama-3. “We strongly consider in open source models,” Arel told VentureBeat. “The more progress shared with the community, the more cool applications and the higher for everybody.”
The potential applications of Tenyx's post-training optimization technology are diverse, starting from creating highly specialized chatbots for specific industries to enabling more frequent incremental updates of deployed models to maintain them current between major releases.
Redesigning the AI landscape
The impact of Tenyx's breakthrough is profound and will level the playing field as firms and researchers gain access to cutting-edge language models without the high costs and limitations related to proprietary offerings. This development could also result in further innovation within the open source community as others look to construct on Tenyx's success.
“It type of raises the query of what this implies for the industry. What does this mean for the OpenAIs of the world?” Arel considered. As the AI arms race continues to heat up, Tenyx's performance in fine-tuning open source models could reshape the AI industry and alter the way in which firms approach natural language processing tasks.
While Tenyx optimized Llama 3 model has the identical limitations as the bottom model, corresponding to: B. occasional illogical or unfounded answers. However, the performance improvements are significant. Arel highlighted the model's impressive progress in math and reasoning tasks, where it achieved nearly 96% accuracy in comparison with the bottom model's 85%.
While Tenyx opens the door to a brand new era of open source AI innovation, the impact of its breakthrough on the AI ecosystem stays to be seen. One thing is definite, nonetheless: Tenyx has shown that open source models can compete with and even outperform their proprietary counterparts, paving the way in which for a more accessible and collaborative future in artificial intelligence.
