Chinese AI startup DeepSeekwhich previously made headlines with a ChatGPT competitor trained on 2 trillion English and Chinese tokens, has announced the discharge of DeepSeek Coder V2, an open-source code language model based on expert knowledge (MoE).
Based on DeepSeek-V2, a MoE model introduced last month, DeepSeek Coder V2 is great for coding and math tasks. It supports greater than 300 programming languages ​​and outperforms cutting-edge closed-source models, including GPT-4 Turbo, Claude 3 Opus and Gemini 1.5 Pro. The company claims that is the primary time an open model has achieved this performance, putting it far ahead of Llama 3-70B and other models within the category.
It can also be noted that DeepSeek Coder V2 maintains comparable performance by way of general reasoning and language skills.
What does DeepSeek Coder V2 offer?
Founded last yr with the mission to “unravel the mystery of artificial intelligence with curiosity,” DeepSeek is a notable Chinese player within the AI ​​race, alongside firms like Qwen, 01.AI and Baidu. In fact, inside a yr of its founding, the corporate has already released quite a few models as open source, including the DeepSeek Coder family.
The original DeepSeek Coder, with as much as 33 billion parameters, performed well in benchmarks with features comparable to project-level code completion and infilling, but only supported 86 programming languages ​​and a 16 KB context window. The recent V2 offering builds on this work, expanding language support to 338 and the context window to 128 KB – allowing it to handle more complex and large-scale coding tasks.
When tested on the MBPP+, HumanEval and Aider benchmarks, designed to guage the code generation, editing and problem-solving capabilities of LLMs, DeepSeek Coder V2 scored 76.2, 90.2 and 73.7 points, respectively, outperforming most closed and open source models, including GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro, Codestral and Llama-3 70B. Similar performance was achieved on benchmarks designed to guage the model's mathematical capabilities (MATH and GSM8K).
The only model that managed to outperform DeepSeek's offering in multiple benchmarks was GPT-4o, which scored barely higher on HumanEval, LiveCode Bench, MATH, and GSM8K.
DeepSeek says it achieved these technical and performance advancements through the use of DeepSeek V2, which relies on its Mixture of Experts framework, as a foundation. Essentially, the corporate pre-trained the bottom V2 model on a further dataset of 6 trillion tokens – consisting largely of code and math-related data from GitHub and CommonCrawl.
This allows the model, which is provided with 16B and 236B parameter options, to enable only 2.4B and 21B “expert” parameters to handle the tasks at hand while optimizing for various computing and application requirements.
Strong performance typically language and argumentation
DeepSeek Coder V2 not only excels in programming and math tasks, but in addition performs well typically logical considering and language comprehension tasks.
For example, it scored 79.2 points within the MMLU benchmark, designed to guage language understanding across multiple tasks. This is a lot better than other code-specific models and almost comparable to the results of Llama-3 70B. For their part, GPT-4o and Claude 3 Opus proceed to steer the MMLU category with scores of 88.7 and 88.6, respectively. GPT-4 Turbo follows closely behind.
The development shows that open coding-specific models are finally excelling across your complete spectrum (not only of their core use cases) and are approaching the latest closed source models.
DeepSeek Coder V2 is now available under a MY Licensesuitable for each research purposes and unrestricted industrial use. Users can download each 16B and 236B sizes in Instruct and Base avatars via Hugging faceAlternatively, the corporate also offers access to the models via API through its platform in response to a pay-as-you-go model.
For those that need to test the capabilities of the models first, the corporate offers the likelihood to interact with Deepseek Coder V2 via Chatbot.
