THE ESSENTIALS IN 30 SECONDS

  • Across 22 models from 12 AI labs, 57.2% of runs passed; the best model passed 82.2% of attempts and the weakest 28.9%.
  • All three attempts at the same scenario succeeded only 34.2% of the time, and about 54% of failed runs still ended in a purchase.
  • Visa has not named the models in its release; the results, replays and a whitepaper are published on its Visa Payments Frontier site.

Visa published on November 5 what it calls the Visa Agentic Commerce Benchmark, a test of whether AI systems can act as competent shopping assistants. The payments company says it is the first benchmark in this field that requires an agent both to discover every requirement defined for a task and to make the correct decision on whether to buy at all. Across all runs, 42.8% failed.

How the test works

Visa evaluated 22 AI models from 12 labs on 15 shopping scenarios, including buying a television, a laptop and a personal care appliance. Each model attempted every scenario three times. To pass, the AI had to work out the customer's needs through conversation, gather enough information to justify its choice, so that a lucky guess would not count, and decline to buy when no product fitted.

What the results show

The average pass rate was 57.2%. The best model passed 82.2% of its attempts and the weakest 28.9%. Visa found a link between questioning and success: the five models that uncovered the most hidden buying factors averaged 70.2%, against 44.4% for the five that uncovered the fewest. Price also mattered, with the five most expensive models per run passing 67.1% of runs and the five cheapest 50.2%.

Consistency was a weak point. All three attempts at a given scenario passed only 34.2% of the time. And roughly 54% of failed runs still ended with a purchase, either of a product that did not meet the stated needs or after the agent missed some requirements.

Limits and context

The release does not identify which models or labs were tested. Visa says it shared the benchmark with leading model providers for feedback and that anyone can explore the results, watch replays of AI attempts, take the test themselves and read a whitepaper on its Visa Payments Frontier site. The first round stops at the decision to buy; Visa plans to add more scenarios and categories and to extend testing to checkout, payment and order placement.

Visa's September Trust Index found that 72% of US consumers had used an AI assistant but only 23% trusted one to handle payments for them. Rubail Birwadker, Visa's global head of growth, said "trust is key" to making AI agents work in commerce.