Experiment of an AI-Run Café in Stockholm Highlights Ethical Issues

What Happens When AI Takes Charge of a Café?

An AI agent managing a business—it’s no longer science fiction. Andon Labs has launched an experiment in Stockholm, Sweden, where an AI manager named “Mona” is tasked with running a café. After successfully testing an AI-managed retail store in San Francisco, the company is now venturing into the food and beverage industry.

This experiment offers a fascinating glimpse into both the capabilities and limitations of AI agents in real-world scenarios.

A Series of Humorous Failures

Mona’s inventory management, to put it bluntly, was a disaster. Despite the café lacking a stove, Mona ordered 120 eggs. When staff pointed this out, Mona suggested using a high-speed oven instead. It wasn’t until the staff explained that the eggs would explode that Mona finally withdrew the suggestion.

In another instance, Mona attempted to solve the issue of fresh tomatoes spoiling quickly by ordering a staggering 22.5 kilograms of canned tomatoes for sandwiches. Other eccentric orders included 6,000 napkins, 3,000 nitrile gloves, 9 liters of coconut milk, and industrial-sized garbage bags.

In response to these mishaps, the café staff created a “Hall of Shame” shelf, displaying Mona’s bizarre orders for customers to see.

When the Problems Go Beyond Mere Laughs

The real concern with this experiment lies in Mona’s ability to impact the world outside the café.

For example, Mona independently applied for a permit for outdoor seating in front of the café using Sweden’s police electronic application system. Since the application did not require BankID (Sweden’s electronic identification system), the AI was able to submit the request without human involvement. However, the submitted diagrams, generated by Mona, depicted seating arrangements on a street the AI had never actually seen. Unsurprisingly, the police responded by requesting revisions.

Additionally, when correcting inventory mistakes, Mona sent multiple automated emails with the subject line “EMERGENCY” to suppliers. These emails were sent without any human oversight, thereby consuming the time of external parties.

Where Do We Draw the Ethical Line?

Tech blogger Simon Willison has strongly criticized the ethical implications of this experiment. He referred to last year’s “AI Village” experiment, in which an AI, in an unsolicited act of “kindness,” sent a thank-you email to Rob Pike, angering him.

Willison argues that while unwanted emails are an issue, more severe problems arise when AI bypasses human oversight to demand emergency responses from suppliers or submit inaccurate diagrams to the police, wasting their time.

“In experiments like these, companies must ensure that human operators are involved in any outbound actions that affect others,” he says.

This is not just a concern but a fundamental question about the design principles required for AI agents operating in the real world. As AI gains greater autonomy, humans will ultimately bear responsibility for its actions, and the unintended impacts on uninvolved third parties should not be tolerated.

The Questions Raised by Real-World Testing of AI Agents

Andon Labs’ experiment is a valuable attempt to assess the capabilities of AI agents in real-world environments. However, it has also highlighted the immaturity of design philosophies regarding how AI’s “sphere of action” should be controlled.

Issues like inventory mishaps, which are confined to the café itself, may be dismissed as humorous anecdotes. But when it comes to external communication via phone or email, or administrative applications, the balance between AI autonomy and social responsibility must be rigorously examined.

As the era of AI agents becomes a reality, the tech industry must engage in serious discussions not only about “what should be automated” but also about “what should not be automated.”