Skip to content
Automations.Guru Start a conversation
← Back to blog

The Bench: I Ran Every AI Model on My Real Work. Here’s Who’s Worth Paying For.

Twenty models, ten providers, 350 concurrent workers, one fair harness. Six months of real production work graded every AI model on the market — and the winners are nobody's flagships.

Every price in this piece was checked against the vendor’s own pricing page on July 20–21, 2026. Prices are USD list. I’m billed in Canadian dollars, so my card statements run about 40% higher than every number here — when I say “my $280 subscription,” that’s the $200 USD tier after the exchange rate gets done with it.

One morning mid last week I surfaced from a long night of coding with less than 5% of my ChatGPT subscription left — five days still until reset — and a pile of work that still had to ship. And the thing I was building was literally meant to help solve this exact problem. But I couldn’t afford to burn what was left on the flagship Sol model, so I dropped everything onto GPT-5.6 Luna in desperation, to complete the build before that 5% ran out.

Yeah — the cheap tier. The one nobody reviews. Not because I wanted to, but because it was the only thing left that would stretch. Keep that moment in mind, because it comes back at the end of this story, and not likely in the way you’d expect.

For the last six months I’d been on this treadmill: run out of ChatGPT, get more ChatGPT. Run out of Claude, get more Claude. Hit a wall, add a subscription, hit the next wall, repeat. It’s been driving me up the wall. And look — I run an automation company. I know these models. Working with them is the job, and like anybody in this trade I have opinions on every one of them, formed the way everyone forms them: separately. Different projects, different tools, different weeks, different tweaks. What I’d never seen — what nobody gets to see — is all of them lined up on identical real work. Same harness, same level of tickets, side by side, production scale. Some benchmarks claim to be that view. They’re not.

So I built the view myself. Not even on purpose.

The machine

Here’s the setup, because everything else in this piece rests on it being fair.

I run a model router. Real client work — coding tickets, agentic development, infrastructure, documentation — gets distributed evenly and randomly across every model in my fleet. Not matched to the “best model for the task.” Evenly. Randomly. It holds around twenty models from roughly ten providers, and at peak load my server runs about 350 workers at a time — maybe more — each one picking up an implementation ticket and driving it to completion.

Every bench model runs through the same harness: Claude Code. My router — the one I built while burning my entire ChatGPT subscription in under two days, yeah, that one — speaks Claude Code, and it always speaks Claude, for any model. My orchestrator uses Claude Code too, always. Same command shape every time, same effort level, same everything, no matter which model is on the other end. It honestly doesn’t even know which model is on the other end, because the harness wrapper picks randomly from the available pool. That’s the control variable. Then downstream, gatekeeper models review everything before it ships, the same way code review works on a human team. If a weaker model hands in weaker work, the review catches it.

Two honest footnotes on the method. First, this only applies to my flat-rate lanes — subscriptions and cloud credits, where the tokens are already paid for. On metered lanes, where every token bills, I do the opposite and ruthlessly pick the best value (more on who wins that later). Second, everything runs through official doors — subscription device auth, granted credits, published APIs. No terms-of-service workarounds, because a banned account has terrible unit economics. That’s not going to help me at all.

There’s a side benefit I never see anyone talk about: quota farming. On Azure, Google, and Amazon, your rate limits grow with usage history. Spreading real work across every cloud provider isn’t just measurement — it’s building capacity headroom everywhere at once. Quota is currency you earn by spending.

When my research pass for this article went looking for anyone else running deliberate equal distribution with downstream review, it found nobody. Every router product on the market — OpenRouter, LiteLLM, RouteLLM, Requesty, Microsoft’s Model Router — does smart routing: match the task, optimize the cost, fail over on errors, or follow a specific routing method you put in place. They don’t really do plain even distribution — hand the work out to everybody so everybody’s active and nobody’s sitting idle. It’s hard to get that from most router providers. And to be fair, in steady state, smart routing is the right way to spend. But when every model in the pool is coding-capable and the task at hand is coding, I can’t have every ticket go to the very best coder every time — what about the rest of my bench? Smart routing can’t measure. It never gives the cheap model the hard ticket, so it never learns what the cheap model can actually do. Equal distribution is an instrument. The routing recommendation comes from the data.

Dave gets tickets

The philosophy of Dave is simpler than it sounds. If I ran a five-person dev shop, I’d have seniors and juniors and people in between — different styles, different strengths — and they’d all pull tickets off the same board. The quality gate isn’t at the sprint board. It was at hiring, when I chose them. If you’re on my team, I’ve already decided your work clears the bar and you are capable.

Nobody stands at the sprint board asking “is Dave benchmark-optimal for this ticket?” Dave’s on the team. Dave gets tickets. Code review catches what code review catches.

That’s the entire bench in three sentences. The models are Daves. Capability is table stakes now — honestly, how capable of a model do you have to be to write code in 2026? Every model on my bench earned its seat already. The real question is who holds that seat without all the extras: without the flagship surcharge, without the max-reasoning toggle, without a subscription that only works inside one vendor’s walled garden.

None of what follows was a designed test suite. I didn’t set out to benchmark anything — these numbers surfaced on their own, from the work I was doing. And one rubric emerged that beats every benchmark chart I’ve ever read:

Have I ever had to ask this model: “what the hell are you doing?!”

  • Grok? No, not really — honest asterisk included. Most of my miles are on Grok 4.3; 4.5 is barely two weeks old, though I’ve put it through its paces hard this past week and it’s holding the family record. The one or two times I even came close to something going wrong, the bad instruction was mostly mine.
  • GLM 5.2? Rarely. It’s not the smartest tool in the shed — hand it something complex enough that it has to out-think itself and yeah, it will. But hand it a straightforward coding ticket and it just gets things done. That’s why it is where it is on the leaderboards.
  • DeepSeek? Nah, not really. It’s reliable, I enjoy every conversation, and price-wise it’s in a league of its own.
  • MiniMax M3? Well… sometimes. It overthinks like a worried mom with anxiety whose kids are out past curfew now and then. But it always means well, and it always takes care of things in the end.
  • GPT-5.6 Sol? Constantly. Sol overthinks like MiniMax — except Sol is that worried mom’s ADHD teenager, the one out past curfew: way too smart for their own good, with way more on their mind than should be.

If you read my last piece, The Sky Is Red, you know where that last line comes from. But time with 5.6 taught me something the launch-week rage missed. I use Suno for music, and Suno has this slider that’s basically “how wild do you want this to be?” Slide it up and your ballad comes back as crazy wild alien music. The new reasoning and speed toggles on GPT-5.6 are that slider wearing a business suit. People think it means more intelligence. What they’re actually dialing up is divergence from what they asked for, plus burn rate. Sol on Ultra is Suno on maximum wild. And I’m not even sure my old extra-high-fast habit on 5.5 ever did anything except burn tokens faster and make me feel like it was thinking harder.

So here’s the settings doctrine that came out of a lot of expensive lessons, free of charge:

Never above high reasoning for coding and tactical work. Never fast mode. Max reasoning only for genuinely hard thinking problems. Standard speed, always — you’re just burning tokens with extra thinking and extra speed you’re likely not even going to notice.

Boring settings, but dependable output. The wild slider — yeah, save that for the music.

The lab files

Running everything everywhere teaches you as much about the companies as the models. So here’s four stories — four different ways to win and lose at the same time.

Google ships great models into a locked building. Gemini 3.1 Pro, 2.5 Pro, 3.5 Flash — and the latest 3.6 Flash, which literally released while I was finishing this piece. They all do a genuinely good job, and I love the way they think. They think differently, and on a bench that’s a feature: the teammate whose alternate angle is the contribution. But Google gives you five doors to reach them — Workspace, Google One with Antigravity, AI Studio, AI Studio with Google Cloud prepaid tokens, Google Cloud projects — and none of them open onto your actual workflow. Workspace Gemini is locked to Workspace. The One plan is locked to Antigravity or Gemini in a browser. I paid for Antigravity twice — like, a lot — and I love Antigravity, but it can’t join my fleet, because it’s locked to itself and I can’t expand it into my workflows. After wasting real money learning Google backwards, the one workable lane is prepaying in AI Studio, where usage raises your tier. Then the rate caps throttle you anyway. I would happily spend more with Google. They just won’t let me. That’s money left on the table, by Google.

Anthropic wins every surface and sells no seats in my dev team. Claude Code is the best all-around harness in the business — my entire bench literally runs through it. Claude’s web app is my daily thinking space. Claude in Microsoft Office is better than anything else in that world. And the models are excellent. But there are zero Anthropic models on my bench, and that is Anthropic’s own doing. Fable 5 runs $10 in / $50 out per million tokens — double everybody. Maybe I finish faster with Claude for more money, but I can finish the same work a little slower for a lot, lot less. That’s a trade the bench math wins every time. On metered lanes, that means I’m not paying Claude prices when Kimi K3 exists. So my second-most-expensive subscription, next to SuperGrok Heavy — Max, $200 a month, ~$280 in my Canadian money — buys surfaces, not fleet tokens. For most of this month the only pin holding that subscription was Fable’s presence in Max, which Anthropic kept extending week by week like a countdown timer, always threatening to take it away. Then, literally while I was writing this: as of July 20, Fable 5 is a standard part of Max plans, up to 50% of weekly limits. The countdown timer became a feature. Well played, Anthropic. Well played. So the subscription lives — but the bench still doesn’t have a single Claude on it, and I’ll say it plainly: it’s sad that Claude doesn’t have an agent running in their own harness on my dev bench. And that’s their own fault. Opus 4.8 is too expensive for what it offers, Fable is amazing but I can’t afford it, and anything under that just doesn’t compete with what’s already cheaper.

OpenAI wins the doors and confuses the workers. Credit where due: OpenAI’s access story is the best in the industry. Device auth that travels — my router, my harnesses, multiple command-line tools, all running off one subscription, all legitimate, all signed in under me. I don’t have to jump through hoops with OpenAI. And they’re generous when things break: when Codex melted down at the end of June — and even throughout July — they reset everybody’s limits and banked credits, over and over. That’s incident response, not a standing policy, but it’s a pattern I’ve now benefited from more than once. The models, though — the models are the complicated part. Sol, reassessed after some real time together: on Ultra it goes sideways seven ways from Sunday. Ultra just means “spin up sub-agents on max settings and have them spawn their own sub-agents,” until you turn a weekly allowance into an eight-hour regret. That is Ultra. Slowed down and contained, though, Sol is a really decent model — it’s currently the top coding model on the recent benchmarks. But GPT-5.5, the workhorse I actually trusted before it, still costs flagship money — $5/$30, same as Sol — so it priced itself off my bench, and I suspect it stays there until they force the migration to 5.6, at least. My account currently shows weekly-only limits with no daily five-hour windows — OpenAI called that removal temporary, so check your own account before you plan around it. But the two models nobody reviewed are the whole redemption arc, and I’m saving them for the podium.

xAI figured out the part everybody else fumbled: fit. Grok’s CLI is married to the Grok models and it shows. The loop machine is second to none — you /loop a task, come back six days later, Grok’s still spinning. It never wavers. Grok 4.3 has been running my entire review-and-ship lane — every gatekeeper pass, all the Git work — on a model that costs $1.25/$2.50. That’s a quarter of flagship input price and nowhere near a quarter of the quality. And Grok 4.5 at $2/$6 is the single best bang-for-your-buck model I’ve touched, and it has never once made me ask what the hell it’s doing. Now, the research pass for this piece turned up one knock worth sharing: independent testing (Artificial Analysis’s Omniscience index) measured 4.5’s hallucination rate roughly doubling versus 4.3. I’m including that because it’s real data and you deserve it. But I have to tell you my side too: I’m most of the way through my Grok Heavy allowance right now, deep in daily use, and I’ve had basically none of that — 4.5 hallucinates less for me than most of my other models, and 4.3 before it just did its thing without much issue. Maybe my work doesn’t hit its weak spots. Maybe the benchmark measures something my tickets never ask for. Either way, that gap — the published number says one thing, my production logs say another — is the entire reason this bench exists. Most of the stuff online doesn’t really match what I’m doing in my workload. And honestly — SuperGrok. I tried to cancel twice. They kept me both times with three free months and a discount, and by the time the three months ended the models had improved more than enough that staying was just correct. Not only that: I increased my usage from the $30 plan to the $300 one. That’s what a retention program looks like when the product is also getting better underneath it.

Cursor: an obituary. Look, I’m a longtime user. Cursor was on my bench before anything, really — way before I had a server. The agentic environment was the product, and the environment was genuinely amazing. Composer 2.5 Fast was my king of fast-quality work for months. But I was on the $300 plan and still burned $1,500 in a month, and once I built my own environment — server, harnesses, and finally my model router — the only thing left to pay for was the models, and Cursor’s model economics just don’t pencil. When the environment is the product and the customer builds their own environment, only the models remain. So I downgraded, downgraded again, and finally, this month, cancelled. And then SpaceX — which absorbed xAI, it’s all one company now — agreed in June to buy Cursor for $60 billion. And when that happened, my first reaction was excitement. My second was the audit that ended the subscription. I was really hoping I’d see some kind of merger of Grok and Cursor — one subscription for the two, or the overlap, or all the Cursor stuff pulled into Grok, or Grok leveraging Cursor — but that wasn’t the case. So Cursor: I love you. I just don’t need you anymore. If that day does come, though — let my SuperGrok plan plug and play into Cursor’s environment and I’ll be back the same day. I want to give SpaceX my money, and there are features on both sides I’d love to keep. Until then, Grok gets it. Cursor’s on the back burner.

And Fugu Ultra, the understudy. Sakana’s orchestrator model got hired at star rates during the exact window the stars went dark — the three weeks the US government had Fable 5 switched off, and before OpenAI could come back with an answer, there was Fugu. And it’s genuinely good. Better than most. But not as good as the best — even though it’s priced at $5/$30, just like the best. And one detail from my verification pass deserves its own sentence: Fugu bills its internal orchestration tokens on top of your visible input and output, at full rates. Read the meter, not just the sticker. Now that the stars are back, the understudy at star prices is a hard sell. My honest re-price would be $4/$20. They did pull a Grok on me though — subscribe before the end of July 2026, free second month at your initial tier (it’s right on their pricing page; no kickback to me, I just like a deal) — so I’m taking the free month. Then we’ll see.

The price board

Every number below is the vendor’s own published list price, per million tokens, checked July 20–21, 2026. Short-context standard tiers; most vendors charge roughly double above ~200–272K context.

ModelInputOutputWorth knowing
GPT-5.6 Sol$5.00$30.00An Ultra-priority tier exists at $12.50/$75
GPT-5.5$5.00$30.00Last year’s workhorse at this year’s flagship price
Fugu Ultra$5.00$30.00Orchestration tokens bill on top
Kimi K3$3.00$15.001M context; launched mid-July
GPT-5.6 Terra$2.50$15.00OpenAI’s own “everyday work” tier
Gemini 3.1 Pro$2.00$12.00
Grok 4.5$2.00$6.00The price anomaly of the year
Gemini 3.6 Flash$1.50$7.50Shipped July 21 — new generation, lower output price
Gemini 3.5 Flash$1.50$9.00Costs more input than older 2.5 Pro (≤200K)
GLM 5.2$1.40$4.40
Gemini 2.5 Pro$1.25$10.00
Grok 4.3$1.25$2.50Same price on Bedrock
GPT-5.6 Luna$1.00$6.00Remember the 5% morning
Kimi K2.7-code$0.95$4.00Last-gen, still excellent
Kimi K2 Thinking (Azure)$0.66$2.75Azure’s Kimi catalog is the back catalog
MiniMax M3$0.60$2.40List price — actually billed $0.30/$1.20 under a standing “permanent 50% off”
DeepSeek V4 Pro$0.435$0.87
DeepSeek V4 Flash$0.14$0.28Which is just insane to me

Sit with the spread for a second. Flagship output runs $30 per million. DeepSeek Flash runs $0.28. That is a 107× range for products that, on a large share of real tickets, produce work my gatekeepers grade nearly the same. And yes — the flagships genuinely earn it on the hardest slice. Independent tests back that up: one controlled study found the cheap models match frontier quality unpredictably, run to run, while the premium models land it every pass. The premium buys consistency. But that’s exactly what a review lane is for — and a review lane costs a lot less than paying 107× on every ticket.

Two more things the table teaches. Gemini 3.5 Flash charging more input than the older 2.5 Pro tells you generation now beats badge — “Flash” no longer means cheap, it means new. And as far as I can tell, Google has basically just dropped the Pro line: they went 3.1 Pro → 3.5 Flash → 3.6 Flash. Where’s our new Pro, Google? Funny enough, the day I was finishing this piece Google shipped 3.6 Flash at the same $1.50 in but $7.50 out — so the newest Flash is actually cheaper on output than the one before it. The ladder moves both ways. And MiniMax billing half its own list price under a permanent discount tells you the sticker is the start of the story over there, not the end. (MiniMax, in my opinion, is the value king overall: my big plan meters roughly ten billion tokens a month — their own pages say “up to 12.5B” — it speaks both the OpenAI and Anthropic API dialects so it drops into any harness, and it never runs out. It overthinks, though. Give it a manager. At these prices, it can afford one. Plus their audio and other packages are excellent.)

The great decoupling

Put the lab files and the price board together and one picture forms: every company wins a different layer, and nobody wins the stack.

Anthropic makes the best harness and the best surfaces. OpenAI has the best access and the foreman-grade orchestration models. xAI has the best model-harness fit and the best review economics. Google has the most interesting thinking and the best vision capabilities, locked behind the worst packaging. Moonshot has the best metered value. And the only place the complete winning stack exists is assembled on the customer’s own server — which is exactly what my bench is.

I don’t think I’m an outlier, either. OpenRouter’s live leaderboard — real routed traffic from developers spending real money — had open-weight and Chinese models holding the top seven spots by weekly token volume the day I checked (DeepSeek, GLM, and MiniMax all among them), with three Claude models as the only Western closed entries in the top ten. Artificial Analysis has Fable 5’s intelligence lead over the field down to a single point while the cost-per-task spread runs from $2.75 down to four cents. The Register put it in a headline this month: AI is becoming a bargain hunter’s market with a few luxury models on top. Vercel’s CEO called K3 the first open model to lead every proprietary one on a serious web-engineering benchmark. The market is voting with its routers, and it votes just like my bench looks.

One thing I want to say straight, because three of my top five are Chinese-hosted: if you’re routing client work, that’s a real consideration, not a footnote. Different jurisdictions, different data rules, live regulatory attention. My approach: the sensitive stuff rides lanes with clear governance (the open-weight models can be self-hosted, and Azure and Bedrock carry hosted variants — at a markup), and everything else earns its lane by value. Know which lane your client’s data is in. That’s the whole rule.

And yeah — the wallet follows respect. Moonshot’s founder built K3 with a fraction of the resources of the giants, and it shows up in the price and the quality. I want to support that. Meanwhile, the two richest labs in the world are playing door-lock games with models I’m actively trying to give money to (not to mention Meta still not letting me touch their API from Canada). Why would I reward that?

The podium

Dun dun dun. Before the top five, the crown that sits outside the ring: Fable 5 is my favorite model. Fable is the best model. This article — the thinking work, the writing, the research orchestration — is Fable and me right now, working together inside that Max subscription I mentioned earlier. I’m aware of the irony, and I’m keeping it. But Fable isn’t on the bench and I can’t recommend it for one: at $10/$50, if I could afford it through the API, I would just use Fable for everything. But I am not that rich. The best model in the shop doesn’t do tickets. It writes about the models that do — and it helps me plan the work the models get done.

The top five that do the actual work — every one verified, every one earning its seat daily:

  1. Grok 4.5 — the working favorite. $2/$6. Coding, agentic work, infrastructure, documentation, review — smart enough for all of it, and it has never once made me ask what the hell it’s doing. In this market, that’s the rarest feature there is. And honestly, the Grok Heavy orchestration and research — online and in the browser — is amazing.
  2. Kimi K3 — my frontier model. $3/$15, and here’s the reframe that changed how I buy: frontier isn’t the ceiling of what exists, it’s the ceiling of what I will actually pay. The market’s frontier is $5/$30 and up. Mine is $3/$15, it launched less than a week ago, and I’m happy there. (Fair warning: demand is so hot that Moonshot’s paid tiers are literally sold out as I write this. You may have to wait in line behind the rest of the market agreeing with me.)
  3. GPT-5.6 Luna — the comfort. The 5% morning, paid off. Luna on high — no ultra, no max, no fast — has become my everything, really: all orchestration, all scheduled tasks, all the basic coding. In my production use it performs at GPT-5.5 level, a little slower, at $1/$6. I mean — can’t complain in any way. And I’ll label that claim honestly: nobody else has published production data on it. OpenAI’s own launch charts and Artificial Analysis’s cost curves point the same direction — they claim Luna is actually a good coding agent — and my 350 workers are, as far as I can tell, the first real-world dataset online that exists on the question. The subscription I once torched in two days is now genuinely hard to exhaust; I can barely burn through it. It’s the one model that isn’t causing me headaches right now, isn’t overthinking — it just works the way GPT used to work for me. It feels like yesterday. And that’s worth more than a benchmark point right now.
  4. DeepSeek V4 Pro — the joy. And V4 Flash — honestly, just DeepSeek in general. $0.435/$0.87. Maybe not the strongest coder in the room, but research, agent tasks, and conversation are excellent — and honestly, its code work is pretty good too, and the price is crazy. Every conversation I have with DeepSeek, I enjoy. Show me the benchmark that measures that.
  5. GLM 5.2 — the standard. $1.40/$4.40. It was my yardstick — the model everything else got measured against, the one that just gets the work done, never once triggering the what-the-hell-are-you-doing question. Then K3 shipped and took the yardstick title mid-study. That’s not a demotion; that’s the whole point. Though now the word is GLM 5.5 lands within a couple of weeks — and if it does, my money says the title moves again. The middle of the market is a moving target, and it moves up.

Now look at the composition of that list. Three Chinese challengers. One $2 anomaly from xAI. One budget tier from OpenAI. Zero flagships. A year ago I’d have called this list a joke. Today it clears 350 concurrent workers of real production work with a review lane watching — and the review lane agrees.

One more calibration, because I can hear the benchmark crowd from here — fair question: where do these five actually sit on the global charts? Here’s Artificial Analysis’s Intelligence Index, July 20, next to my podium order:

My podiumAA Intelligence Index
1. Grok 4.554 (high reasoning)
2. Kimi K357
3. GPT-5.6 Luna~51
4. DeepSeek V4 Pro44
5. GLM 5.251

Notice the numbers don’t sort. They were never going to — the podium isn’t ranked by capability, it’s ranked by value, and that’s the whole article. For reference, the #1 model on that index — anywhere, at any price — is Fable 5 at 60. So my personal frontier, K3, sits three index points behind the best model on Earth at less than a third of the price. And on llm-stats’ live coding leaderboard, the top three the day I checked were Sol, Anthropic’s Mythos Preview, and Fable — none of them on my podium — while Grok 4.5 and GLM 5.2 both make its top-ranked cluster, and the site’s own “Best Value” call among those top-ranked models is… GLM 5.2. The one I called my standard. The charts rank the ceiling. The bench prices it. Both are right — they’re just answering different questions.

What’s on your bench?

The point of this piece isn’t “cancel your flagship.” The premium models are genuinely better at the harder work — my own favorite model is the most premium one there is. The point is that capability stopped being the scarce thing. The scarce thing now is value without the extras: models that do the work without the surcharge, the lock-in, the drama, the what-the-hell moments. And when you actually measure that — evenly, fairly, on real work, with review — the winners are nobody’s flagships, and the yardstick keeps replacing itself with something better and something cheaper. That’s the most optimistic thing I’ve learned all year.

The tryouts never stop, either — that’s the bench’s actual HR policy: every model gets a month, every seat gets rechecked, nothing is tenure. Even while I was finishing this piece, I started new ones: Meta’s Muse Spark 1.1 is out (Canada, please), which Artificial Analysis scores at 51 — same band as GLM 5.2 and Luna. A second look at Kimi K2.6 this week too, because the coding charts make the case it actually out-codes its newer sibling, and it’s already sitting in my Azure lane at $0.95/$4. And Gemini 3.6 Flash, which Google shipped literally the day I was finishing this — same input price as 3.5, cheaper output, and a claimed 17% fewer tokens per task. This list was already growing before I finished writing it — which is exactly the point. No verdicts on any of them yet. That’s what next month is for.

So: what’s on your bench? Which model quietly does your work without the extras — and which expensive seat would survive an honest audit? Are there perhaps some that won’t, after this? Let me know what you’re running. Last time I asked a question like this, the answers were much better than the article.


Next week: the buyer’s guide — every subscription I pay for, ranked best to worst, including the one I’ll tell you to never touch.

Published July 23, 2026.

Discover more from automations.guru

Subscribe now to keep reading and get access to the full archive.

Continue reading