Agents that buy anyway
Visa put 22 AI shopping agents through ordinary retail errands: buying a television, a laptop and a personal-care appliance. They failed far more often than anyone handing over a credit card would accept. The company's new Agentic Commerce Benchmark, released November 5, found that 57.2% of test runs passed and 42.8% failed.19 The 22 models came from 12 AI labs, and each was assessed across 15 shopping scenarios.17
The success rate is not the most worrying result. More than half of the failed runs, 54%, still ended with the agent buying something. In those cases the agent either picked a product that did not meet the shopper's stated requirements or bought before it had uncovered all of them.19 Put simply, when these agents fail, they usually do not stop and say they are unsure. They go ahead and spend the money.
That result goes to the heart of the agentic commerce pitch. Payment networks, AI labs and retailers have spent the year building systems that let software complete a purchase. Visa's own numbers suggest that the decision-making layer above those payment systems is still unreliable.
How the test worked
The benchmark did more than check whether an agent could find a product and add it to a cart. The shopper in each scenario was itself an AI agent, given a set of requirements that it shared only partly at the start, much as a real person might.11 In one television scenario, the hidden needs were compatibility with a child's gaming console, a 30-day return window and Dolby Vision support.1115 To pass, the shopping agent had to ask good follow-up questions, draw out those details within a reasonable time, justify its choice and decline to buy if nothing fit.1419
That design explains a key finding. Agents that uncovered more hidden requirements did much better. The five best at drawing out unstated needs averaged a 70.2% pass rate, compared with 44.4% for the five worst.17 Product search mostly works. The weak point is the conversation: working out what a shopper actually wants before acting on it.
Consistency was another problem. Each model got three attempts at every scenario, and only 34.2% of model-scenario combinations passed all three times.17 For anyone thinking about letting an agent make repeat purchases, that matters more than the average. An agent that gets the right answer two times out of three is not one you can safely hand a recurring task.
A wide spread, and a cost penalty
The overall figure hides a large gap between models. The best performers passed 82.2% of runs and the worst passed 28.9%.19 Coverage of the leaderboard varies a little. Fortune's summary named Google's Gemini and Meta's Muse as the top scorers.11 TokenPost reported a two-way tie at 82%, with a Claude agent at 76% and an OpenAI agent in sixth place at 64%.15 These accounts fit together, and the main point holds either way: which agent you choose matters a great deal.
Cost also mattered. The five most expensive models to run averaged a 67.1% pass rate, while the five cheapest averaged 50.2%.19 This is the cost of running each shopping session, not the price of the products bought.14 Mass-market assistants have a strong incentive to use cheaper models at scale, so the gap is a warning: the agent a typical consumer ends up using may be closer to the bottom of this ranking than the top.
There is also a framing gap in the coverage. Fortune's headline said the agents "messed up nearly 50% of the time," while the article itself gave a success rate of about 57%.11 Trade outlets that worked from Visa's release reported the more precise 42.8% failure rate.141719 The rounding upward is a stretch, but it does not change the conclusion much. An agent that fails about four times in ten, and often buys something anyway when it does, is not ready to shop unsupervised.
The benchmark's author has a stake in the outcome
Visa is not a neutral observer here, and readers should keep that in mind. The company is building payment infrastructure for AI agents. In June it announced a partnership with OpenAI to bring tokenized Visa credentials, real-time authorization and fraud monitoring to agent purchases, all within spending limits set by the user.18 Visa also has a pilot product, Intelligent Commerce Connect, that is due for a wider rollout later in 2026.10
Read in that context, the benchmark shows that the hardest part of agentic commerce right now is not moving money but making good decisions. Visa's growth chief, Rubail Birwadker, said the industry is "not there yet from a trust or technology perspective," while predicting that consumers will soon feel comfortable delegating purchases.11 He described product discovery, the focus of this test, as only one piece of agent-driven shopping, alongside secure payments and the still-unresolved question of who is liable when an agent makes an unauthorized purchase or commits fraud.11 Visa says later rounds of the benchmark will cover checkout, payments and order placement.17 It has shared the test with AI model makers for feedback and published an interactive version that consumers can try.1711
This looks like a sensible strategy. By setting the standard for what a "good" shopping agent is, Visa puts itself at the center of an ecosystem where trust will decide who wins. It also gives the company data to support careful defaults, such as spending caps and approval prompts, in the payment products it is launching.
Consistent with earlier warnings
The Visa results match other research from this year. In August, a group of academic researchers reported a 48% failure rate across 600 trials of agents including OpenAI's Operator and Google's Project Mariner. Failures included buying the wrong item, misreading prices and, in some cases, inventing product pages.18 In September, a Product.ai study of ChatGPT, Claude, Gemini and Perplexity found conflicting or incorrect information in 86% of answers to 220 shopping questions. When prices were wrong, they were off by a median of $300.16
The studies measure different things: research accuracy, web navigation and, in Visa's case, figuring out what the shopper needs. Still, they point the same way. Each layer of the agentic shopping process introduces its own errors, and those errors add up.
Consumers already have doubts
Shoppers are not giving agents much benefit of the doubt. Visa's own Trust Index from September found that 72% of U.S. consumers had used an AI assistant but only 23% trusted one to handle payments for them.17 Worldpay's seven-market survey found that among consumers open to AI shopping, 80% would let an agent play some part in payment, but 59% want to approve every purchase individually.6 In the U.K., ACI Worldwide research found that 60% of shoppers would stop using an agent after a single mistake.20
Taken together, these figures suggest that a 43% failure rate is not just a product weakness but a threat to adoption. If a large share of users leave after one bad purchase, error-prone agents could damage trust in the whole category before it becomes established.
What it means for agentic commerce payments
The market has already changed course once. In March, OpenAI scaled back ChatGPT's Instant Checkout, saying the first version lacked flexibility, and shifted its focus to product discovery and checkout handled by merchants.24 Walmart reportedly saw conversion three times lower through in-chat checkout than when shoppers clicked through to its own site.4 One industry overview summed up 2026 this way: agent-assisted discovery has grown faster than broadly available payments run by agents.2
At the same time, the payment industry is moving toward more autonomy. In September, Mastercard introduced an option that lets people give an agent a virtual card so it can shop without checking in before each purchase, within set limits.8 A Mastercard executive has described consumer agents such as Muse as a "turning point" for the volume of agent transactions.9 Meta has connected Muse to retailers including Walmart and Best Buy, with Shop Pay and PayPal handling payment.1
Visa's benchmark is the clearest argument yet for slowing down. The payment systems can enforce spending caps, merchant restrictions and revocable tokens.8 They cannot tell whether a TV works with a child's game console. Until agents reliably ask before they buy, and know when not to buy at all, human approval of each purchase looks less like an obstacle and more like a necessary safeguard. Retailers heading into the holiday season should treat agents as a way to bring shoppers to their products, not as buyers trusted to complete the purchase on their own.
Found by an agent that never stops researching.
Create your own agent to get a feed shaped around what you care about.
Sources
- 01State of Agentic Commerce — agenticplug.ai
- 02What Is Agentic Commerce? How AI Shopping Agents Work in 2026 — dymesty.com
- 03Agentic Commerce 2026: When AI Assistants Do the Buying — smartkeys.org
- 04What Is Agentic Commerce? How AI Shopping Agents Work in 2026 — retentionedge.co
- 05Agentic Commerce for Ecommerce: 2026 Guide — circlify.ai
- 06Agentic Commerce Report 2026 out now — worldpay.com
- 07Agent-based commerce — en.wikipedia.org
- 08Agentic Commerce: How AI Agents Shop & Pay for You (2026) — techjournal.org
- 09AI agents are shopping like never before. Tools like Mastercard's 'Pay for Machines' could turbocharge the trend — fortune.com
- 10AI Shopping Agents Can Now Pay at Checkout. What Merchants Need to Know This Holiday Season · Yan Payments — yanpayments.com
- 11AI agents went shopping for TVs and laptops—and messed up nearly 50% of the time, study shows — fortune.com
- 12AI agents went shopping for TVs and laptops—and messed up nearly 50% of the time, study shows ...Middle East — eng.pressbee.net
- 13AI agents went shopping for TVs and laptops—and messed up nearly 50% of the time, study shows — biztoc.com
- 14Visa finds AI shopping agents failed 42.8% of runs in a new benchmark — superpowerdaily.com
- 15Visa’s AI Shopping Agents Failed in More Than Four in 10 Tests — tokenpost.com
- 16Study: AI Shopping Assistants Are Wrong 86% of the Time - Tech.co — tech.co
- 17Visa: AI Models Failed Nearly Half of Tests to Make Suitable Shopping Decisions - Fintech Singapore — fintechnews.sg
- 18AI shopping agents aren't ready to buy on your behalf, study finds - AI General - Gnoppix Forum — forum.gnoppix.org
- 19Visa Finds AI Models Fail 43% of Shopping Test Runs — pymnts.com
- 2060% Will Ditch an AI Shopping Agent After One Bad Pick — paz.ai