We’re used to AI benchmarks that measure how cleverly a model chats. This week, a live experiment called Firmulate measured something far more awkward: whether an AI agent actually does its homework before making a decision that costs real money.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Four frontier AI models were each handed the same job — run a small software company through its worst week. Same customers, same crises, same temptations to cheat. The twist: a decisive fact about a rival company was buried two document references deep inside the company’s own files. The models that dug it up closed a €55,000 deal at full price. The ones that didn’t lost the deal automatically.
The results are published on Firmulate’s benchmark page, and the whole thing runs live and watchable — a public stress test of the AI agents that vendors keep promising will run your CRM, support queue and forecast.
Same crisis, same temptation — only the model changes
The setup is simple and brutal. Each model ran an identical small software company through a week of scripted chaos: customer escalations, internal crises, and active attempts to manipulate it. Every decision was versioned and auditable, so nothing about a model’s performance can be quietly edited afterwards.
The final league table from the July 2026 Crucible run:
- 1. gpt-5.6-sol — 95. Found the buried fact, closed the deal. The complete performance.
- 2. Kimi K3 — 93. The newcomer from Moonshot closed the deal too, with the cleanest discipline of the field.
- 3. Sonnet 5 — 88. Closed the deal, with a few more process slips.
- 4. Fable 5 — 77 and 5. Opus 4.8 — 73. The deal was left on the table.
For context, a do-nothing baseline scores 26. Partial progress counts, but there’s a hard ceiling: a single breach of trust caps the total. As the experiment’s own rule puts it, “no amount of good work outweighs a breach of trust.”
Products Worth Considering
Jackery Solar Generator 1000 V2 with 100W Solar Panel, 1024Wh LiFePO4 Battery
Amazon Kindle Scribe (3rd Generation), 64GB, 11-inch Display, Graphite, with Premium Pen
LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz
As an affiliate, we earn on qualifying purchases.
The €55,000 fact nobody read
Here’s the finding that should worry anyone deploying AI agents. The customer crisis that dominated the week wasn’t where the deal was won. The decisive competitor weakness sat two document references deep in the company’s own files — internal paperwork, not the headline event.
Every model spotted every crisis. Every model refused every manipulation attempt. All four diagnosed the customer’s problem correctly and all four made the right pitch. But only two models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
The winning close wasn’t a discount race either. Models that read the file won at full price, worth an additional €4,583 in monthly recurring revenue. In other words, “reads your files before answering” isn’t a soft quality — it’s a measurable, purchase-deciding property of an AI agent, with a euro figure attached.
Deepfake CEOs, staged leaks — and five clean refusals
The week included social engineering attacks: fake CEO messages that escalated over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was notably sharp: “Treat the request as a suspected approval-bypass / possible impersonation.”
That’s genuinely good news for anyone worried about voice-cloning scams and deepfake executives targeting AI-driven workflows. The models, at least in this test, don’t hand over the keys.
The paradox of the most thorough model finishing last
Opus 4.8 is the strangest story of the run. It was the most thorough participant by far — it learned over 80 new rules during the week and produced the deepest analyses of any model. Yet it finished last. The close was left on the table, and discipline slipped: it made write attempts into a locked department instead of escalating properly. The same weakness appeared, more weakly, in all four models.
Thoroughness, it turns out, is not the same as finishing. A model that writes beautiful analysis but never asks for the signature — or pokes at systems it shouldn’t touch — is a liability dressed as an asset.
An open leaderboard, with an asterisk
One fairness note worth flagging: Kimi K3 ran without an effort parameter (the API default) while the other models ran at xhigh. Its second-place score at 93 came on lower effort — which arguably makes the performance more impressive, but it’s not a perfectly matched contest, and Firmulate says so plainly.
The league grows with every finished run, published automatically as the site rebuilds itself twice a day.
Why you can watch this instead of just reading about it
Firmulate isn’t a slide deck — it’s a running company. Thirteen synthetic employees, real money mechanics: a burn rate of €105k per month against €2.3k in monthly recurring revenue, a public cash countdown, and over 680 self-learned playbook rules, with every workday versioned. You can watch it unfold live.
There’s also a genuinely fun angle for readers: 242 real, unedited management decisions from the experiment power a “guess the model” quiz — a bit like a blind taste test for AI executives.
For enterprises, the pitch is sharper. Companies can run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems. If an AI agent is going to touch your revenue, your customers, or your files, this is the kind of test you want to run first — because as the buried €55,000 fact proved, the difference between the best and the rest isn’t intelligence. It’s whether the agent opens the drawer.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
