Visa has put the shopping-agent hype to a real test, and the results are sobering. The payments giant launched its first-of-its-kind Agentic Commerce Benchmark on November 5, sending 22 AI models from 12 labs through 15 realistic shopping challenges — buying a TV, a laptop, or a personal care appliance. The models passed just 57.2% of their runs, and in more than half of the failures they bought something anyway, even when it did not match what the customer asked for.
## The Test
The benchmark is built to catch the failure that matters in real shopping: the customer rarely knows what they want. Each model must talk with the shopper, uncover the hidden requirements — budget, compatibility, a return window, a gaming console that has to work with the TV — and then either buy the right thing or buy nothing at all.
Every model got three attempts at each scenario, and consistency proved to be the weak spot. All three attempts passed just 34.2% of the time, so a model that aced a shopping trip once often stumbled on the repeat. The five models best at drawing out hidden requirements passed 70.2% of their runs; the five worst passed only 44.4%.
## The Leaderboard
The best models passed 82.2% of their attempts and the worst just 28.9%, with a surprise at the top of the leaderboard. Google’s Gemini Aragon 4 and Meta’s Wonder tied at about 82%, a Claude agent came third at 76%, and an OpenAI agent finished sixth at 64%, according to Fortune.
Price helped but did not decide the outcome. The five priciest models passed 67.1% of their runs, against 50.2% for the five cheapest — a real gap, but not a guarantee. Visa’s message is that discovery, not spending power, is the skill that separates the models that shop well from the ones that guess.
## Buyer Beware
The most worrying number is what happened after a failure. About one in two failed runs — 54% — still ended in a purchase, either the wrong product or one bought before the agent had uncovered everything the customer needed. That turns a bad recommendation into a charge, a refund, or a dispute.
## Our Read
Visa is not grading homework here; it is trying to write the rules of a market it intends to sit in the middle of. Every payments network now wants to be the trust layer for agentic commerce — [Mastercard has its own Agent Pay programme](https://geroundtable.com/mastercard-agent-pay-for-machines-early-access/) — and a benchmark only Visa runs is also a standard only Visa controls.
That is not a criticism: somebody has to set the bar, and Visa’s test is genuinely demanding. The more interesting question is whether the labs accept the verdict. A benchmark that flatters nobody is useful; one the labs quietly ignore is just marketing, and Visa says it has shared the results with providers for feedback.
The failures are about judgement, not raw capability. [Axiom’s agents recently verified 7,654 pages of pure mathematics](https://geroundtable.com/axiom-agents-verify-stacks-project/), so the models can plainly do hard, sustained work. Shopping asks something different: reading a person who does not fully know what they want, and knowing when to stop.
## Why It Matters
Agentic commerce only works if shoppers trust an agent with their money, and right now they mostly do not. Visa’s own Trust Index, released in September, found that while 72% of US consumers had used an AI assistant, only 23% trusted AI to handle payment transactions for them.
Agents are already taking on real jobs for real people — [Grok’s bot now haggles on the phone](https://geroundtable.com/grok-bot-phone-calls/) and negotiates bills — so the trust gap, not the technology, is the bottleneck. Until an agent can be relied on to buy the right thing, or buy nothing, shoppers will keep the wallet to themselves.
## Background
Visa says it will extend the benchmark to checkout, payment, and order placement, and add more shopping scenarios and product categories. The benchmark sits alongside Visa Intelligent Commerce, the company’s portfolio of products, programs, and partnerships for agentic commerce. Anyone can try the test themselves on Visa Payments Frontier, where Visa has published the full results and a whitepaper.
Source: [Visa’s announcement](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html) and [Fortune’s report](https://jbiznews.com/ai-agents-went-shopping-for-tvs-and-laptops-and-messed-up-nearly-50-of-the-time-study-shows/)
0 Comments