Every organisation running AI in production has had this meeting. Finance shows a chart. The bill is up again. Someone points out that a smaller model costs a fraction of what you are paying, and asks the obvious question.

And the engineer in the room says the only responsible thing they can say, which is: I do not know if it is good enough.

That is not obstruction. It is accurate. The evidence usually available is a handful of prompts someone tried in a playground, and a benchmark score measured on work that has nothing to do with yours. Neither tells you whether your customers will notice. So the safe answer is to change nothing, and the bill goes up again next quarter.

The cost problem is real. The blocker is not cost. It is that quality is asserted rather than measured, so switching is a bet nobody wants their name on.

Measure it on your own traffic

The traffic you actually serve is the only test set that matters. It has your customers' phrasing, your edge cases, your unglamorous long tail of requests nobody wrote a test for.

So use it. Keep serving production from the model you trust, and send the same work to the candidate alongside it — same input, second model, no effect on what your customer receives. Squidder calls that a shadow run. Compare the answers: not once, on a cherry-picked example, but continuously, on whatever real traffic arrives, for as long as you need to stop arguing.

At the end you are not holding an opinion. You are holding a rate. The answer has this shape:

6 days · 4,180 sampled requests
equivalent           96.2%
candidate worse       2.9%
candidate better      0.9%
cost delta            -71%

That number is either high enough to act on or it is not, and either way the meeting is over in five minutes.

Why this changes the conversation, not just the number

The engineer can now say yes. They are not being asked to vouch for a model on instinct. They are being shown its behaviour on the company's own work.

The finance owner gets a date. "We will know in two weeks" is a different answer from "we would have to look into it".

A negative result is also a result. Sometimes the cheap model is not good enough, and knowing that cheaply is worth having. You stop relitigating it every quarter and you stop feeling vaguely guilty about the bill.

The reason to build the habit rather than run it once is that the same comparison answers every other change you are nervous about. Can we move this workload in-region. Can we drop to the smaller context window. Does the new version of the model we already run regress anything. Each is the same experiment with a different candidate, and an organisation that can run it stops being afraid of changing its mind.

What this does not do

Comparison costs something. You are paying for the candidate's work as well as the real one for the duration. It is a fraction of what you would save and it is not free, so run it on the traffic that matters and stop when you have your answer.

Automated judgement is not perfect judgement. Comparing two answers for equivalence is itself a model's opinion, and on subtle work you should read a sample yourself. The value is not that the judgement is infallible. It is that you are looking at thousands of real cases instead of the four somebody pasted into a chat window.

Where to start

Pick the workload with the highest volume and the lowest stakes. It is almost always something internal: a summariser, a classifier, an enrichment step. Nobody will be harmed if it gets slightly worse, and it is probably a meaningful share of your bill.

Run the comparison for a week. Then either make the change with a number behind it, or close the question with evidence and stop paying it attention.

Both outcomes are better than the one you have now, which is an open question nobody can answer and a bill that grows while it stays open.