The takeover of interoperability standards reminiscent of the model context protocol (MCP) can provide firms an insight into the functioning of energetic ingredients and models outside their walled limits. However, many benchmarks don’t record real interactions with MCP.
Salesforce AI Research developed a brand new open source benchmark, which calls MCP university, which is presupposed to pursue LLMs because they interact with MCP servers in the true world, and argue that they may use a greater picture of real and real-time interactions of models with tools enthusiasts. In his first tests it found that models like Openai'S recently GPT-5 published Are strong, but still don’t work so well in real scenarios.
“Existing benchmarks mainly give attention to isolated points of LLM performance, reminiscent of: a paper.
MCP-University records the model output through tool use, multi-gymnastics tool calls, long context windows and huge tools. It is predicated on existing MCP servers with access to actual data sources and environments.
Junnan Li, Director of AI Research at Salesforce, told Venturebeat that many models are “still suspended restrictions that they hold back within the event of a company quality tasks”.
“Two of the best are: long context challenges, models can lose an outline of knowledge or, when coping with very long or complex inputs, struggle against reason,” said Li. “And unknown tool challenges, models are sometimes unable to make use of unknown tools or systems seamlessly in the best way wherein people can adapt. For this reason, it just isn’t crucial with a DIY approach with a to follow the person model for power agents, but to depend on a platform that mixes data context, enlarges and trust Guardrails to satisfy the needs of Enterprise. “
MCP university joins other proposed benchmarks with MCPlike for instance MCP radar From the University of Massachusetts Amherst and the Xi'an Jiaotong University in addition to the Beijing University of Posts and Telecommunications' McPworld. It also builds on McPevals that Salesforce released in July and mainly focuses on agents. According to Li, the largest difference between MCP University and McPevals is that the latter is assessed with synthetic tasks.
How it really works
MCP-University evaluates how well each model explains quite a few tasks that imitate the tasks undertaken by firms. Salesforce said that MCP University comprises six ceremony homes utilized by firms: location navigation, repository management, financial evaluation, 3D design, browser automation and web search. For a complete of 231 tasks, 11 MCP servers.
- The location navigation focuses on geographical considering and the execution of spatial tasks. The researchers have chopped the Google Maps MCP server for this process.
- The repository management domain deals with code base operations and establishes a connection to the Github MCP to uncover version control tools reminiscent of repo search, output tracking and code processing.
- The financial evaluation combines with the Yahoo Finance MCP Server to guage quantitative argument and decision -making for financial market.
- The 3D design evaluates using computer-aided design tools via the MCP blender.
- The browser automation connected to MCP of dramatists tests the browser interaction of the browsers.
- The web search domain uses the Google Search MCP Server and the Fetch MCP to envision the seek for open-domaine information and is structured as a more open task.
Salesforce said that it needed to design latest MCP tasks that reflect real applications. For each domain, they’ve created 4 to 5 sorts of tasks that the researchers imagine that LLMS can easily do. For example, the researchers have assigned a goal to the models that included route planning, identified optimal stops after which found the goal.
Each model is rated how you might have done the tasks. Li and his team opted for an execution-based evaluation paradigm and never for the more frequent LLM-AAA-Judge system. The researchers found that the LLM-as-a-Judge paradigm “just isn’t well fitted to our MCP university scenario, since some tasks are designed for using real-time data while the knowledge of the LLM judge is static.”
Salesforce researchers used three sorts of evaluators: Format evaluators to find out whether the agents and models meet the format requirements, static evaluators with a purpose to evaluate the correctness over time and dynamic rating for fluctuating answers reminiscent of flight prices or github problems.
“MCP-University focuses on the creation of demanding tasks with an execution basis that may test the agents in complex scenarios. In addition, MCP-University offers an expandable frame/code code base for the structure and evaluation of agents,” said left.
Even the large models have problems
To test MCP-University, Salesforce rated several popular proprietary and open source models. This includes GROK-4 of XaiPresent Anthropic'S-Claude-4 Sunt and Claude 3.7 Sunt, Oenais GPT-5, O4-Mini, O3, GPT-4.1, GPT-OTP, Google'S Gemini 2.5 Pro and Gemini 2.5 Fkash, GLM-4.5 from WouldPresent Moon shotKimi-K2, QwenQWEN3 CODER and QWEN3-235B-A22B-Instruct-25507 and Deepseek-V3-0304 Deepseek. Each model tested had at the least 120b parameters.
In his tests, Salesforce found that GPT-5 had the perfect success rate, especially for financial evaluation tasks. GROK-4 followed and hit all models for browser automation, and Claude-4.0-sun rounds off the highest three, even though it not recorded higher performance figures than the next models. GLM-4.5 was the perfect under open source models.
However, MCP University showed that the models had difficulty handling long contexts, especially for location navigation, browser automation and financial evaluation, with efficiency significantly decreasing. The moment the LLMS meet unknown tools, their performance also drops. The LLMs showed difficulties to do greater than half of the tasks that firms normally perform.
“These results show that the present border LLMS remains to be reliable to reliably perform tasks in various real MCP tasks. Our MCP University Benchmark due to this fact offers a difficult and needed test bed for the evaluation of LLM performance in areas which are under tracked by existing benchmarks,” said the paper.
Li said Venturebeat that he hopes that firms will use MCP-University with a purpose to achieve a deeper understanding of where agents and models fail with tasks in order that they will improve their framework conditions or the implementation of their MCP tools.
