Integrate · harden · meter · evidence
That starts with getting it into the systems you already run and keeping it standing on a bad night, because nothing that falls over has a margin. Then the routing and the model mix, so the bill is defensible line by line — and the baseline and test set that answer the question either way. Even when the answer is: don’t scale this. No bench here needs the next phase to exist.
The situation
Industry evidence cannot tell you whether your deployment worked. A baseline taken from your own systems can.
Why your own baseline is the instrument
There is no industry number you can hold your own deployment against, and it is worth knowing why. The one randomised trial that measured whether AI made experienced developers faster found it made them slower, and that the developers believed the opposite both before and after. Its authors ran it again later and found no significant effect either way, and they say even that probably understates any gain, because developers who would not work without AI dropped out; they now call their own new data only very weak evidence and are redesigning the study, and what fills the gap meanwhile is a self-report survey they tell you themselves to treat sceptically.∗3
So an outside number can tell you what happened to somebody else’s deployment, and that is all. Yours is a measurement, and it has to be taken on your systems.
Something you can check today, without us
A prompt shorter than your model’s caching minimum does not get cached, and it does not raise an error either — it just quietly bills at full rate. A cache hit costs a tenth of the input price, so a hot path sitting under the minimum is the cheapest thing in your stack to find and the easiest to miss. Look at your longest-running prompt template, count its tokens, and compare it against the minimum for the model you are on. That is a five-minute job and it is yours, not a deliverable.
The gap
Both lines are indexed to their own first period, so one scale carries them and neither is flattered by a second axis. Nearly every programme can draw the first line from its own invoices. The second needs an owner and an instrument. The line here is flat because nothing measured it, which is not the same as nothing having happened.
The baseline
Ninety days from your own history and a holdout left untouched, with your finance owner agreeing in writing what the metric means. Look at the points either side of the line: you cannot tell by eye whether anything moved, and that is the whole problem. If your change has already shipped, the work starts from whatever history your systems already hold — and the next change is the one worth getting in front of.
The last mile
The step that comes apart is the one reaching into something only your company has: the internal service nobody documented, the scheduler that predates the team maintaining it. Iteration caps so it stops. Retries that resume from the last durable state instead of starting over. A person in the loop before anything privileged happens. None of it is glamorous, and all of it is the difference between a pilot and something you can leave running.
The unit
What one resolved case genuinely costs, once inference, the retries nobody logs, and the person who checks the output are all counted. Drawn in equal parts on purpose, because which one dominates is what the work has to find out rather than something to assume in advance. What the number licenses is yours to decide, and sometimes it licenses stopping.
What we do
Named scopes with an end state you can check, in the places a platform vendor cannot reach for you: your own systems, your own cost structure, your own definition of correct. This is where the work starts rather than a closed catalogue — if what you need sits next to one of these, say so.
One system at a time, through your change control, with no edits to anything live: the internal service nobody documented, the scheduler that predates the team maintaining it. The job is to shape it into a tool surface an agent can actually use — not to pour the whole API into a context window, and not to modernise your estate.
Iteration caps so it stops. Retries that resume from the last durable state instead of starting over. Checkpoints you can resume from, and a person in the loop before anything privileged happens. The difference between a demo that works and a run you can leave alone overnight.
A cost baseline per workflow, then the work that moves the number: finding the hot paths that silently fail to cache, heavy fan-out moved onto the cheapest model that still passes, batch where latency does not matter, and an escalation policy that says when a bigger model is worth paying for.
What we would set up: a ninety-day baseline from your own export, a holdout left untouched, and your finance owner agreeing in writing what the metric means before anything changes. Then a test set for the behaviour you care about, versioned with the system, and a write-up of what moved, what did not, and what remains unproven.
How it works
You describe what you have deployed, where it stopped, and what you are being asked to justify. We tell you plainly whether any of this helps, and what it would take. No charge, and no obligation.
If it is worth doing, we scope it to what we found on the call — named deliverables, a fixed duration, and the limits of what it will prove, agreed in writing before it starts. One engagement at a time, which is why the scopes stay narrow.
Everything lands in your repository, documented for the people who will keep it running, and built so you can take it out again. There is nothing here that only works while we are still being paid.
Where your data sits. Work is deployed in your own cloud account, so your existing agreement with that cloud provider governs the data.
Where it stopped
The first pass is not a strategy document, it is a short list of things to look at in an order that finds the cheapest answer soonest. Pick the one that sounds like your week, or describe it in your own words. Nothing is sent anywhere and no model is called.∗2
Illustration. Not a diagnosis of your system.∗2
Idle
Nothing is sent anywhere. The reply is already in this page.
Where the money actually goes
An agent run costs many times a single chat turn, and the cost is rarely spread evenly — one step usually dominates. Spend tagged per workflow and per step, not per API key.
Fan-out that nobody is waiting on does not need the interactive path, and usually does not need the largest model either.
Cost per resolved unit of work, in the unit your finance owner already reports. Tokens are not that unit, which is why the current figure does not travel.
An engagement here ends with a spend baseline in your finance owner's own unit, and a ranked list of what to change first.
What happens on the bad night
A run that restarts from the top after a transient failure pays for everything twice, and often repeats its side effects too. Interrupt it mid-run and watch where it comes back.
The first place to look is the step calling something only your company has. Iteration caps, a timeout, and a durable checkpoint before it.
Placed in front of the irreversible action, not in a review queue behind it.
You end with a run that survives being interrupted, and a written record of which failure modes it is now covered against.
Why it cannot reach your systems
A tool that returns forty fields to answer a two-field question spends the context you needed for reasoning. The surface is the engineering; the connector is the easy part.
Scoped, delegated and revocable, in place before the first write rather than after the first incident.
Read it, wrap it, and keep the wrapper honest about what it cannot do. This is the part no platform vendor can ship for you.
You end with the integration in your own repository, documented for whoever keeps it running.
What it is allowed to do
A per-agent workload identity federated from the directory you already run, rather than a shared key with a long life. Scoped, and revocable without resetting anything for a human.
The tools it can call, the records it can write, and the actions that stop for a person first. A capability list is a design artefact, not something you discover during an incident.
Untrusted retrieved content marked and kept separate from instructions, output validated against a schema rather than trusted, tools behind least-privilege tokens the app owns, and the same cases re-run in your CI so a prompt change cannot quietly undo it.
An engagement here ends with a credential you can revoke, a written capability list, and an exportable record of what each agent actually did.
What the board can actually be shown
A few dozen real cases with checkable outcomes beats a thousand synthetic ones, and it is the part that needs your staff rather than ours.
Agreement with your humans is the number that matters. An uncalibrated judge is an opinion with a decimal point.
In your CI, on a score drop, before anyone has to argue about it in a meeting.
You end with a baseline, a test set, and an answer that holds whichever way it comes out.
That one needs a conversation
Five scenarios are written into this page and yours is not one of them, so anything it produced here would be a guess dressed as an answer.
Describe where it stopped in an email and a person reads it, and you get a real reply rather than a generated one.
Or break one on purpose.
The first check above is “where it cannot resume”. Here is a run with a checkpoint after the third step and a person in front of the seventh. Interrupt it whenever you like and watch where it comes back from.
Nine steps. A checkpoint after the third, a person before the seventh. Nothing has run yet.
Get in touch
Three lines is enough to start: what you deployed, where it stopped, and what you are being asked to justify. What comes back is either a time for the call, or a straight answer that this does not help — including when the thing you need already ships in a product you own, or comes free with the model you are on.
You will be dealing directly with the person who does the work, and nobody else takes it over halfway through.