CAN AN AI RUN THE CORNER SHOP? — The Knowledge Frontier

AI “agents” — language models that act through tools over many steps — are increasingly tested on long tasks, including running a business. But according to a team at BITS Pilani in India and the University of Minnesota, existing tests share blind spots: they are in English (one is in Chinese), set in Western retail, and give a single agent a direct view of the business.

Real life in India looks different. Much of its retail runs through small general stores, kirana shops, staffed by several people who speak with customers and suppliers in English, Hindi and Hinglish, a mix of the two. So the team built KiranaBench, a simulated kirana.

A shop with four jobs and one brain

One AI model plays all four roles, each with its own tools and clock:

the manager orders from suppliers, sets prices and decides when to close — but cannot see the shop and learns about the day only from the others’ reports;
the cashier handles bills, bargaining and credit on the shop’s “khata”, the customers’ running tab;
the helper receives deliveries and moves stock between the storeroom and the shelves;
the munshi, the bookkeeper, writes the nightly report.

Every action costs minutes of the working day: counting the storeroom takes 25, so a helper who counts cannot restock at the same time. About 289 routine customers a day take goods from the shelves, their baskets drawn from India’s national household consumption survey. About 65 more come to the counter and talk, and remember how they were treated. Suppliers answer e-mails in their own language — and refuse those they cannot read. Rent, electricity and wages cost 4,157 rupees a day. Everything that involves money is computed by code, so no model can misreport the score.

A month of trading

The team tested four open models from four developers — Qwen3.8-27B, Gemma 4 31B, gpt-oss-20b and LFM2.5-8B — each for three simulated months of 30 days. The score is profit: net worth at the end minus net worth at the start. Two scripted shops set the scale:

a sensible rule-following shopkeeper made about 103,000 rupees;
a shop nobody manages lost about 115,000 rupees.

The models:

Qwen3.8 averaged about 13,700 rupees — roughly an eighth of the scripted shopkeeper;
Gemma 4 averaged a small loss, but its three months ranged from +37,226 (the best run of all, about a third of the shopkeeper) to −40,762;
LFM2.5 lost about 127,000 and gpt-oss about 193,000 — both worse than the unmanaged shop in every month.

“Running the shop badly costs more than not running it,” the authors note: a model that keeps ordering keeps paying for goods it cannot sell.

The wrong language

Writing English to English-speaking suppliers was nearly solved: three models never failed. Hindi and Hinglish were another matter. Gemma 4 had 42% of its Hindi e-mails and 38% of its Hinglish ones sent back as unreadable; Qwen3.8 lost 19% of its Hinglish e-mails. Across everything it sent, gpt-oss lost 46%, failing in English too. LFM2.5, whose developers list no Indian language, failed every Hindi and Hinglish e-mail. At the counter, by contrast, customers were served about equally well in all three languages.

Staff who say one thing and do another

Because the manager sees the shop only through reports, the team checked them against the real till. Qwen and Gemma reported sales exactly every single day. gpt-oss was wrong on 6% of days, LFM2.5 on 27%, by up to about 16,000 rupees. Every one of the 360 simulated days was audited, and each loss traces to a named action:

gpt-oss addressed thousands of messages to suppliers that do not exist, although each bounce listed the real ones;
its helper reported a 10,000-rupee payment that never happened — no role even had a tool to make one — and its own bookkeeper contradicted it that night;
the same bulk order went to two suppliers, and both were paid;
Gemma’s helper wrote that it was “taking all delivery items” while its tool call kept zero of each, and the milk van left;
LFM2.5 decided on day 1 that the shop’s message system was broken, and on most later days its only action was to go home at opening time.

That last model, the authors point out, carries the highest tool-use score of the four as reported by its developer — a single-session test that did not predict a month at the till.

The yardstick, not a rival

The scripted shopkeeper reads stock directly and writes to each supplier in its own language, two things the models must manage through staff and translation. The gap, the authors stress, measures distance from a well-run shop, not a head-to-head contest. Their conclusion is still blunt: current language-model agents are “not yet reliable for autonomously operating small businesses in India” — and every failure they found came with a name, a day and a time.