Companies storm AI agents into production – and lots of of them will fail. But the rationale has nothing to do with their AI models.
On the second day of VB transformation 2025The industry leaders shared hard-won lessons from the usage of AI agents on a scale. One by Joanne Chen, General partner with moderated panel Foundation Capital, contain Shawn Malhotra, CTO at Rocket companyuse the agents on the complete apartment of the apartment from the mortgage insurer to the client chat; Shailesh Nalawadi, product manager at Birdthe agent scorer experiences for corporations construct up in several verticals; and Thys Waanders, SVP of the AI transformation at perceptionwhose platform automates customer experiences for giant corporate contact centers.
Their joint discovery: Companies that construct up evaluation and orchestration infrastructure are first successful, while those that storm with powerful models for production fail on a scale.
>> See all of our transformation 2025 reporting here
The ROI reality: beyond easy cost reduction
An essential a part of the success broker for the technical AI is to know the Return on Investment (ROI). Early AI agents focused on the fee reduction. While this stays a key component, company managers now report more complex ROI patterns that require different technical architectures.
Cost reduction wins
Malhotra shared essentially the most dramatic example of rocket corporations. “We had an engineer (the) capable of construct an easy agent in about two working days to resolve a really area of interest problem that is known as” transmission tax calculations “within the mortgage insurance section of the method. And two days of effort save us a million dollars a 12 months,” he said.
For the perception, Waanders found that the prices per call are a key metric. He said when AI agents are used to automate parts of those calls, it is feasible to shorten the common handling time per call.
Sales methods
Saving is one thing; Getting more income is different. Malhotra reported that his team had improvements: Since customers receive the answers to their questions faster and have a great experience, they convert with higher installments.
Proactive income options
Nalawadi emphasized completely recent revenue skills through proactive public relations work. His team enables proactive customer support and changes before the shoppers even recognize that they’ve an issue.
An example of the delivery of food shows this perfectly. “You already know when an order will come too late and as an alternative of waiting for the client to bother and call them, you possibly can see that there was a possibility to realize it,” he said.
Why do AI agents break in production
There are solid ROI opportunities for corporations that provide the AGENT -KI, but there are also some challenges in production provision.
Nalawadi identified the core technical failure: Companies construct AI agents without evaluation infrastructure.
“Before you even start constructing, it’s best to have an Eval infrastructure,” said Nalawadi. “We were all previously. Nobody is committed to producing without unit tests. And I feel a quite simple way of serious about Evaly is that it’s the Unit test to your AI agent system.”
Conventional approaches for software tests should not suitable for AI agents. He noticed that it is solely impossible to predict all possible inputs or to write down comprehensive test cases for interactions for natural language. The Nalawadi team learned this through customer support provisions in retail, food and financial services. Standard quality security approaches were missing from Edge that occurred in production.
Ki -Test -KI: The recent paradigm for quality assurance
What should organizations do given the complexity of AI tests? Waanders solved the test problem through simulation.
“We have a function that we are going to soon publish to simulate potential conversations,” said Waanders. “So there are essentially AI agents who test AI agents.”
The tests should not only tests of conversation quality, but additionally the behavioral evaluation on a scale. Can it help to know how an agent reacts to offended customers? What about several languages? What happens when customers use slang?
“The biggest challenge is that you just don't know what you don't know,” said Waanders. “How does it react to something someone could provide you with? You only find it out by simulation of conversations by really sliding it under 1000’s of various scenarios.”
The approach tests demographic variations, emotional conditions and outskirts that human QA teams cannot treat.
The upcoming complexity explosion
Current AI agents do individual tasks independently. Company leaders have to organize for a distinct reality: Hundreds of agents per organization learn from one another.
The implications of the infrastructure are massive. When agents share and work together, the error modes multiply exponentially. Conventional monitoring systems cannot follow these interactions.
Companies now should do architects for this complexity. Retrofitting the infrastructure for multi-agent systems costs considerably greater than correct creation from the beginning.
“If you quickly go forward in a company that’s theoretically possible, you’ll learn tons of of you, and perhaps you’ll learn from one another,” said Chen. “The variety of things that might occur explodes. The complexity explodes.”
