Meta has Approved the most recent entry in its Llama series of open generative AI models: Llama 3. More specifically, the corporate has unveiled two models in its recent Llama 3 family, with the remaining set to release at an unspecified future date.
Meta describes the brand new models – Llama 3 8B, which incorporates 8 billion parameters, and Llama 3 70B, which incorporates 70 billion parameters – as a “big leap” in comparison with the previous generation Llama models, Llama 2 8B and Llama 2 70B . Performance clever. (Parameters essentially define an AI model's capabilities for an issue, akin to analyzing and generating text; models with higher parameter counts are generally more powerful than models with lower parameter counts.) In fact, Meta says that for his or her respective parameter counts, Llama 3 8B and Llama 3 70B – trained on two purpose-built 24,000 GPU clusters – are are amongst probably the most powerful generative AI models available today.
That's quite a claim. How does Meta support it? Well, the corporate points to Llama 3 models' results on popular AI benchmarks like MMLU (which attempts to measure knowledge), ARC (which attempts to measure skill acquisition), and DROP (which tests a model's reasoning based on paragraphs of text tests). As we've written before, the usefulness – and validity – of those benchmarks is up for debate. But for higher or worse, they continue to be one among the few standardized ways AI players like Meta evaluate their models.
Llama 3 8B outperforms other open models akin to Mistral's Mistral 7B and Google's Gemma 7B, each of which contain 7 billion parameters, on not less than nine benchmarks: MMLU, ARC, DROP, GPQA (a set of biology, physics and chemistry) related questions ), HumanEval (a code generation test), GSM-8K (math word problems), MATH (one other math benchmark), AGIEval (a problem-solving test set), and BIG-Bench Hard (a standard sense assessment).
Now, Mistral 7B and Gemma 7B aren't exactly innovative (Mistral 7B was released last September), and in among the benchmarks Meta cites, Llama 3 8B performs just a number of percentage points higher than each. But Meta also claims that the Llama 3 model with the larger variety of parameters, Llama 3 70B, is competitive with flagship models for generative AI, including Gemini 1.5 Pro, the most recent model in Google's Gemini series.
Photo credit: Meta
Llama 3 70B beats Gemini 1.5 Pro in MMLU, HumanEval and GSM-8K, and while it could possibly't compete with Anthropic's strongest model, Claude 3 Opus, Llama 3 70B performs higher than the second weakest model within the Claude 3 series . Claude 3 Sonnet, on five benchmarks (MMLU, GPQA, HumanEval, GSM-8K and MATH).
Photo credit: Meta
Meta has also developed its own test set that covers use cases starting from coding and inventive writing to reasoning to summarizing and – surprise! – Llama 3 70B prevailed against Mistral’s Mistral Medium model, OpenAI’s GPT-3.5 and Claude Sonnet. Meta says it denied its modeling teams access to the set to take care of objectivity, but provided that Meta developed the test itself, the outcomes should after all be taken with a grain of salt.
Photo credit: Meta
On a more qualitative level, Meta says users of the brand new Llama models will experience more “controllability,” a lower likelihood of refusing to reply questions, and greater accuracy on quizzes, questions related to history, and STEM fields akin to engineering and science, as well general programming should expect recommendations. This is thanks partly to a much larger data set: a group of 15 trillion tokens, or a staggering ~750,000,000,000 words – seven times the dimensions of the Llama 2 training set. (In AI, “token” refers to subdivided bits of raw data, just like the syllables “fan,” “tas,” and “tic” within the word “improbable.”)
Where does this data come from? Good query. Meta wouldn't say, only revealing that it got here from “publicly available sources,” contained 4 times more code than within the Llama 2 training dataset, and that 5% of that set contained non-English data (in about 30 languages) that’s being improved needed to perform in languages apart from English. Meta also said it used synthetic data – AI-generated data – to create longer documents for training the Llama-3 models. a somewhat controversial approach as a consequence of possible lack of performance.
“While the models we’re releasing today are optimized for English editions only, the increased data diversity helps the models higher recognize nuances and patterns and deliver strong performance across quite a lot of tasks,” Meta writes in a shared with TechCrunch Blog post.
Many providers of generative AI view training data as a competitive advantage and due to this fact keep it and the associated information at their fingertips. But details about training data are also a possible source of mental property lawsuits, one other incentive to disclose rather a lot. Current reporting revealed that in its effort to maintain up with AI competition, Meta once used copyrighted e-books for AI training, despite warnings from the corporate's lawyers; Meta and OpenAI are the topic of an ongoing lawsuit by authors, including comedian Sarah Silverman, over the providers' alleged unauthorized use of copyrighted data for training.
So what about toxicity and bias, two other common problems with generative AI models (including llama 2)? Does Llama 3 improve in these areas? Yes, claims Meta.
Meta says it has developed recent data filtering pipelines to enhance the standard of its model training data and that it has updated its two generative AI security suites, Llama Guard and CybersecEval, to try to forestall the misuse and unwanted text generation of Llama 3 models and others. The company can be releasing a brand new tool, Code Shield, designed to detect code from generative AI models which will introduce security vulnerabilities.
However, filtering just isn’t foolproof – and tools like Llama Guard, CyberSecEval and Code Shield only go thus far. (See: Lama 2's tendency to do that Inventing answers to questions and disclosing private health and financial information.) We'll should wait and see how the Llama-3 models perform within the wild, including testing by scientists using alternative benchmarks.
Meta says the Llama 3 models – which at the moment are available for download and support Meta's Meta AI assistant on Facebook, Instagram, WhatsApp, Messenger and the online – will soon be hosted in managed form on quite a lot of cloud platforms including AWS, Databricks, Google Cloud, Hugging Face, Kaggle, IBM's WatsonX, Microsoft Azure, Nvidia's NIM and Snowflake. Versions of the models optimized for hardware from AMD, AWS, Dell, Intel, Nvidia and Qualcomm may even be available in the longer term.
The Llama 3 models might be widely available. However, you’ll notice that we describe them as “open” quite than “open source”. That's because despite Meta's claims, his Llama model family just isn’t as non-committal as one can be led to imagine. Yes, they can be found for each research and business applications. However, meta forbids it To prevent developers from using Llama models to coach other generative models, while app developers with greater than 700 million monthly users must apply for a special license from Meta, which the corporate will grant – or not – at its discretion.
More powerful Llama models are on the horizon.
Meta says it’s currently training Llama-3 models with a size of over 400 billion parameters – models with the power to “communicate in multiple languages,” absorb more data, and understand images and other modalities in addition to text, which is what the Llama -3 series would produce consistent with open releases like Hugging Face's Ideas2.
Photo credit: Meta
“Our goal within the near future is to make Llama 3 multilingual and multimodal, have longer context, and further improve overall performance on core (large language model) features akin to reasoning and encoding,” Meta writes in a blog post. “There’s rather a lot more to come back.”
As a matter of fact.
