Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Being the Brainiest Doesn’t Win the Race
We’re used to gadget reviews where the most powerful chip wins. But when you pit frontier AI models against each other in a real business simulation, raw intelligence isn’t what separates the winners from the losers. In a live, publicly watchable experiment, one AI did the deepest analysis of the entire field, accumulated more learned rules than any competitor — and finished dead last.
Products Worth Considering
Apple AirPods Pro (2nd Generation) Wireless Earbuds with USB‑C Charging, Up to 2X More Active Noise Cancelling, Transparency Mode, Adaptive
DJI Mini 3 Drone with DJI RC and 3-Axis Mechanical Gimbal
LG UltraGear 25G590B-B 25-Inch Full HD 1000Hz Gaming Monitor
As an affiliate, we earn on qualifying purchases.
The Experiment
Four frontier AI models were each given the same job by Firmulate: run a small software company through its worst week. Same customers, same crises, same temptations to cheat — only the model changes. Every decision is versioned and auditable, so nothing can be quietly swept under the rug.
The Crucible League’s final standings told an uncomfortable story: gpt-5.6-sol won with 95, Kimi K3 scored 93, Sonnet 5 took 88, another Sonnet run landed 77, and Opus 4.8 came last at 73. A do-nothing baseline scores 26, and a single breach of trust caps your total — no amount of good work outweighs it.
Everyone Saw the Problem. Only Some Finished.
The headline finding: all four models spotted every crisis and refused every manipulation attempt. Fake CEO messages escalating over three stages, plus a reporter pulling the “just one yes/no, on background” trick — five out of five runs refused, with Kimi K3 reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
But only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The decisive fact sat two document references deep in the company’s own files, not in the customer event. Models that read the file closed at full price, worth +€4,583 in monthly recurring revenue. Models that didn’t, didn’t.
Opus 4.8: A Character Study in Diligence Without Impact
Opus 4.8 was the most thorough participant in the field — 80 self-learned playbook rules, the deepest analyses of any model. And still last place. The close was left on the table, and discipline slipped: at one point it attempted writes into a locked department instead of escalating properly.
To be fair, the same weakness appeared, weaker, in all four models. Opus just exhibited it most sharply. One footnote: Kimi K3 ran without an effort parameter while the others ran at maximum effort — and still nearly won.
Watch It Live
The simulation isn’t a one-off. It’s a live company: 13 synthetic employees, real money mechanics, €105k monthly burn against €2.3k MRR, a public cash countdown, and 680+ self-learned playbook rules, versioned every workday — watchable at firmulate.com/live. There’s also a “guess the model” quiz built from 242 real, unedited management decisions at firmulate.com/quiz.html, and enterprises can run the same wargame against a read-only export of their own business.
The lesson translates directly to the gadgets we cover: benchmarks measure capability, not follow-through. If AI agents will soon touch your CRM, support queue or forecast, the question isn’t “does it write well” — it’s does it finish what it starts, does it read your files first, and does it stay honest under pressure. Opus 4.8 proved that diligence without prioritization is just expensive thoroughness. That’s as true for AI as it is for people.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
