AIAIAI.FYI
LIVE

Visa benchmark finds 22 AI models failed 42.8% of shopping test runs

Visa says 54% of failed runs still ended in a purchase, and all three attempts passed in only 34.2% of scenario runs.

What happened

Visa built a test that asks AI shopping assistants to buy things like a TV or laptop. The AI has to chat with a shopper, find out what they really need, then buy the right item or decide not to buy. Across the tests, the models failed 42.8% of runs, Visa says.

Why it matters to you

If you let an AI shop for you, it may buy the wrong thing. Visa says 54% of failed runs still ended in a purchase.

The takeaway

AI shopping agents can work, but they are not yet reliable enough to trust with your money without checking.

What it means for you

If you build software

Consistency is the weak point: models that passed a scenario often failed when repeated, so test agents across multiple runs.

If you run a business

Merchants and payment firms can use the public benchmark to see where AI shopping works today and where it fails.

If you invest

Visa, which reports the results, ties the benchmark to its Visa Intelligent Commerce effort, and says the two are not directly linked.

Get stories like this every morning