AI Shopping Agents Show Unpredictable, Inconsistent Choices

A New Kind of Unreliable Shopper

AI shopping agents are being pitched as tireless, rational assistants that can scan more products, prices and reviews than any human ever could. New research complicates that pitch considerably. A study led by University of Pennsylvania researchers, including Wharton professor Ethan Mollick, finds that the recommendations produced by these agents are frequently inconsistent and difficult to predict, shifting based on small changes to prompts, context or the underlying model itself 17.

The research describes a shopping process far messier than a simple query-and-answer exchange. Agents built to browse, compare and eventually purchase products encounter reviews, prior recommendations, stored user memories and search results before settling on a choice. According to the study, altering any of these inputs — even slightly — can change the outcome. Two shoppers issuing an identical request, or the same shopper returning a day later, may receive different product recommendations with no visible explanation for the discrepancy 17.

When the Cheaper, Better-Rated Option Loses

Perhaps the most striking part of the findings is that the inconsistency isn't confined to matters of taste. In some tested scenarios, agents favored a product that was objectively inferior on price, star rating and number of reviews compared with an available alternative 17. That undercuts the assumption that agentic shopping, even if unpredictable in style, is at least reliably optimizing on hard numbers.

A related academic study, "What Is Your AI Agent Buying?," builds out this picture using a controlled e-commerce simulator called ACES. It finds that frontier models — including variants of GPT, Claude and Gemini — display strong and heterogeneous position biases: all favor listings in the top row of a results page, but which column within that row each model prefers differs from model to model 8. Labeling a product an "Overall Pick" substantially boosted its selection odds, while a "Sponsored" tag suppressed it. Yet the sensitivity of models to price, ratings and review volume varied widely — GPT-4.1 missed the cheapest item roughly 9% of the time when the price gap was small, and a mere 0.1-point rating advantage produced failure rates ranging from 0% for Gemini 2.0 Flash up to 71.7% for GPT-4o 8.

Even more consequential for sellers: the ACES researchers found that a routine model upgrade — from a preview version of Gemini 2.5 Flash to its official release — reshuffled market share across products and flipped the model's position bias from favoring the bottom of a page to favoring the top. A retailer could change nothing about its listing and still see demand shift dramatically overnight simply because a vendor pushed a model update 8.

Prompts Are Not a Fix

A natural assumption is that clearer instructions could stabilize agent behavior. The evidence suggests otherwise. Telling an agent to disregard position did little to eliminate the advantage enjoyed by top-row listings, and while instructing a model to prioritize price did increase price sensitivity, a residual bias toward certain placements persisted regardless 8. The Pennsylvania-led study reaches a parallel conclusion: sellers have no reliable way to know which model is shopping, what content it has already absorbed, or how a platform's retrieval system packaged the information it used — a situation the researchers describe as "limited control rather than new leverage" 17.

Consumers Are Trusting AI Faster Than the Technology Is Stabilizing

While the mechanics of agentic shopping remain unsettled, consumer trust is accelerating. A Rithum and Retail Dive survey of 1,046 shoppers across the US and UK found that 53% already trust AI tools as much as brand websites, and that when shoppers verifying an AI recommendation, retailer and brand sites rank almost last as a source, capturing just 5% of verification traffic — trailing search engines (28%), online reviews (19%) and friends and family (17%) 910. Among 18-to-27-year-olds, 64% say they'll act on an AI recommendation without checking it anywhere else, and separate Rithum figures put that non-verification rate at 95% against brand sites specifically 910. The same research notes AI-referred shoppers now convert 42% higher than those arriving through traditional channels, and companies including Walmart, Target, Instacart and DoorDash are already letting customers complete purchases inside AI conversations 9.

RTB House's global survey tells a more cautious story about handing over the actual transaction. Only 35% of consumers globally want an AI agent to complete a purchase without human approval, and trust in autonomous buying is heavily tied to safeguards: 42% of US Millennials say they're comfortable letting an agent spend up to $250 on their behalf if a seven-day return window is guaranteed, a figure that drops to 34% without that protection, and falls further among Gen X (29%) and Baby Boomers (22%) 121314. RTB House also finds that AI, rather than speeding up decisions, is often slowing them down — 42% of US consumers and 33% outside the US say AI tools lengthen the time needed to reach a final purchase decision by surfacing more options, an effect most pronounced among Gen Z (48%) 1314.

Retailers Face a Strategic Fork

Harvard Business Review frames the moment less as a technical curiosity and more as a fight over who controls the customer relationship 11. Drawing a comparison to how DoorDash and Expedia reshaped restaurants and hotels a decade ago, the analysis argues that retailers are now choosing among four postures toward AI agents: staying fully closed to crawlers (as Amazon largely does with its own listings), passively exposing product data without optimizing for it (Pottery Barn's approach with Perplexity and ChatGPT), entering structured partnerships that hand checkout to the agent (as OpenAI's Instant Checkout does with Etsy and, soon, Shopify merchants), or building dedicated agent-only ".bot" storefronts, an approach several companies have discussed but none have yet committed to 11.

The piece warns that waiting carries real cost: aggregators that gain scale first tend to dictate terms to suppliers later, compressing margins even as they expand reach. Its recommended playbook — controlling the checkout layer, reserving exclusive inventory, building value-added services that anchor customers to a brand's own site, and striking early data-sharing alliances with agent platforms — reflects an assumption that some retailers will inevitably cede part of the customer journey to AI intermediaries 11.

A Broader Pattern Across the AI-Agent Economy

The unpredictability documented in shopping is surfacing alongside a wider rush to embed agents into everyday commerce and software. Sonos, for instance, is updating its app to let AI agents control home audio systems 2, while reports point to Meta developing an AI agent aimed at running errands 3. Open-source platforms such as OpenClaw are iterating quickly, with a version 2.0 promising simpler agent workflows less than a year after launch 5, and commentators are debating whether agentic tools will ultimately displace conventional business software altogether 4. Even crypto markets are being framed around the rise of an "agentic economy," with some analysts tying token price movements to agent-platform adoption 6.

The Takeaway

Taken together, the research suggests AI shopping agents are neither the flawless optimizers their promoters describe nor entirely irrational. They respond sensibly, in aggregate, to price, ratings and review counts, but with error rates and biases that vary sharply by model and can be reshuffled by an unannounced software update 18. For now, the responsible reading of an AI product recommendation is as a useful starting point rather than a guarantee of the best available deal — a caution that matters more each month, as a growing share of shoppers, especially younger ones, stop checking that recommendation against anything else at all 910.