Production: latency, reliability, and the ops you already know

A production request is a path, not a model call. Authenticate the user. Authorize the corpus. Retrieve. Build the prompt. Call the model, or a cheaper model first to classify 'is this in scope.' Check schema. Log metadata. Render. If any hop fails, you already know where the user goes, because you designed the fallback before you needed it. Fallback is part of the happy path. An incident is a surprise. A fallback is a design.

If the model is down: cached answers for FAQs you actually see, keyword search only with the chunks on the screen, or hand to a human with the question pre-filled. If quality is low — eval drop, schema fail rate up — shrink to retrieve-only: show the chunks, do not generate. Users prefer an honest search result to a stuck spinner or a fluent lie. Write those three degraded modes down. Pick which one fires on which signal.

Routing is ops, not architecture theatre. Cheap small model for classify and 'is this in scope.' Frontier for the actual answer when it is in scope. You designed this on day 5. Production is making it real with a flag, a timeout, and a metric on how often you escalate. If every request goes to the frontier model, you do not have routing. You have a bill.

Timeouts need a number. First token in under two seconds is a different promise from complete in under four. If the vendor is slow, fail to fallback rather than wait twelve seconds and then fail. Users experience the wait as 'it does not work.' Your availability SLO will look fine while adoption dies. Measure what the user waits, not only whether the API returned 200.

Caching is a control on both latency and cost, and a risk on freshness. Cache FAQ answers with a short TTL and a corpus version key so a SOP change invalidates them. Do not cache personalized or ACL-sensitive completions unless the cache key includes the user identity. A shared cache is how you leak. Put cache on the diagram with its key and its TTL, or do not cache.

The path must fail closed on authz. If identity is down, you do not retrieve 'as a default user.' If the index is down, you do not skip retrieve and let the model talk from weights. Those two failure modes are how you get fluent unauthorized answers. Prefer a visible error or a retrieve-only fallback that also cannot retrieve. Closed is better than clever.

Measure each hop or you will argue about the model when the index is cold. Retrieve time, route time, generation to first token, generation to complete, schema check, render. A delivery lead who can say 'retrieve is 1.8 seconds of the 4.2' can assign work. End-to-end only is a blob. Tracing is not optional on a path you claim to operate. Put hop timings on the same dashboard as the SLO, even if v1 is a log you grep.

Retries turn a timeout into a bill and a duplicate write. Retry reads with a cap. Do not retry writes without idempotency keys on tool calls. If the model timed out after it had already opened the ticket, a retry opens two. You already know this from payment integrations. Apply it on day one of tools, not after the first duplicate. Read-only v1 still needs a retry cap so a hung vendor does not multiply spend while the user waits.

Availability is whether the path responded, not whether the answer felt wise. The API, your app, retrieve, and the model hop — or the fallback you counted as success. Write the number: 99.5 percent of requests get a designed response (answer, refuse, or fallback) inside the latency SLO. 'It is usually fine' is not an SLO. An SLO is a number you will freeze changes for.

Latency is two numbers: p95 to first token, and p95 to complete. Streaming makes first token the thing users feel. Complete is what you need for schema checks and for writes. Internal copilot, eighty users: first token under two seconds, complete under four is a reasonable starting bar. If you cannot hit it, reduce context, route, or cache. Do not ask users to wait because the prompt is luxurious.

Quality is a rolling score on a golden sample, not a single thumbs-up percentage. Reuse the day 16 bar: groundedness and recall on a weekly sample, must-refuse still passing. Thumbs can sit next to it as a tripwire. Thumbs alone will lie — users upvote fluent wrongness and downvote honest refusals. If quality burns the error budget, you freeze prompt changes. That sentence is how you stop a well-meaning engineer from tweaking production every afternoon.

Cost is dollars per successful task, with a daily cap alarm. Token burn is a trailing indicator. Task cost is the one finance understands and the one that catches 'we retrieve seventy chunks and call the frontier model twice.' Set a daily cap that pages a human. A cap without a page is a number on a slide. You already know bill shock from cloud. This is cloud with worse defaults, because every request can get more expensive by accident of prompt length.

Error budget is the miss you tolerate before you freeze. If availability or quality or spend burns the budget for the week, prompt changes stop, retrieve experiments stop, and you spend the remaining time on reliability or on rolling back. This is the same governance you use on a mature service. Apply it before the service is mature, or it will never get there.

Write four SLOs, not twelve. Availability, latency, quality sample, daily spend. Owners. Freeze rule. You can add more later. A long list nobody manages is how SLOs become wallpaper. Put them on the same page as the dashboard tiles so the number and the graph live together.

Error budget is the miss you tolerate before you freeze prompt and retrieve changes. If quality or spend burns the budget for the week, you stop decorating the prompt and you spend the remaining time on reliability or rollback. This is how you stop a well-meaning engineer from tweaking production every afternoon. Write the freeze rule next to the four SLOs. A number without a freeze is a wish. You already run this on mature services. Start here before the service is mature, or it never will be.

Availability counts designed responses: answer, refuse, or fallback inside the latency SLO. It does not count 'the user liked it.' A fallback that fired correctly is a success against the availability SLO and a mark on the fallback-rate tile. That split matters. If you count fallback as failure, people will disable fallback to keep the graph green, and users will get spinners. If you hide fallback, you will not know the model has been down for two hours. Count both, separately.

Nondeterminism means the same prompt can yield different tokens. Temperature, sampling, and the model itself. For a copilot, you usually want temperature near zero on factual answers, and you still will not get bit-for-bit replay. Do not promise replay. Promise a bar on groundedness and a pin on the model. Sponsors who grew up on deterministic systems need this sentence once, clearly, without a lecture on probability.

Latency budgets get eaten by retrieve, by long context, by tool loops, and by the model. Measure each hop. If you only measure end-to-end, you will argue about the model when the index is cold. A delivery lead who can say 'retrieve is 1.8 seconds of the 4.2, generation is 1.1, the rest is our app' is a delivery lead who can assign work. That sentence is why tracing is not optional.

Streaming is a user-experience control as much as a technical one. First token on screen tells the user the system is alive. For confirm-on-write, you may not want to stream into a send. For Q&A, you almost always do. Decide per action. Do not stream a customer email into a text box a hurried human might send halfway through.

Retries are how you turn a timeout into a bill and a duplicate write. Retry reads with a cap. Do not retry writes without idempotency. If the model timed out after it had already opened the ticket, a retry opens two. You already know this from payment integrations. Apply it. Put idempotency keys on tool calls.

What you tell a sponsor: we do not promise the same sentence twice. We promise the path is up, the answer is grounded to a bar, the refuse cases refuse, and we will freeze changes if those bars burn. We promise a fallback when the model is down, not a spinner. We promise a pin so a vendor surprise is a scheduled change. That is an ops story a CIO can hear without translating.

What you do not tell a sponsor: 'the model is nondeterministic so we cannot have SLOs,' or 'we will just look at thumbs,' or 'latency depends.' Those sentences abdicate the job. Uncertainty is real. Unmanaged is a choice.

Temperature near zero on factual answers, and you still will not get bit-for-bit replay. Do not promise replay. Promise a pin, a groundedness bar, and a freeze if the bar burns. Sponsors who grew up on deterministic systems need that sentence once, clearly, without a lecture on probability. Write it in the SLO sheet in language they can reuse in their own steering. If they hear 'it depends' they will hear 'unmanaged.'

Streaming is a user-experience control as much as a technical one. First token on screen tells the user the system is alive. For Q&A, stream. For confirm-on-write, do not stream into a send box a hurried human might fire halfway through. Decide per action, the same way you decided HITL per action. A single streaming default across the product is how you get a half-written customer email in a text box with a send button. Turn that default off on writes.

Vendors deprecate. New versions change tone, eval scores, latency, and cost. Pin a version. When you move, run the golden set, compare cost and latency, canary a percentage, keep a rollback pin. Write this as a change calendar item. Do not let 'the default model' float under you. Floating latest is an unscheduled release with no owner.

Prompt and model are a pair. Changing both on the same day is how you cannot tell what broke. Change one, run the set, then the other if you must. The change ticket names which one moved. This is ordinary isolation of variables. Production prompt editing plus a silent model bump is how a week of debugging becomes folklore.

Canary means a slice of traffic, not 'Priya will try it.' Five or ten percent of requests, or one team, with the same eval pulse and the same dashboard tiles, for a named period. If quality or latency or cost moves the wrong way, you roll back the pin. Canary without a rollback is a partial outage you chose.

Who approves a model move is the same list as who waives a quality miss, because a model move can cause a quality miss. Put it on the change calendar with the golden-set result attached. A vendor email that says 'please migrate by the 30th' is a trigger, not an approval. You already treat vendor-forced upgrades this way in other platforms. Do it here.

Document the pin where operators can find it. Config, not a comment in a notebook. The incident page from yesterday should say where the pin lives and how to revert it. If reverting takes a deploy window you cannot hit at 21:00, you do not have a rollback. You have a hope and a pipeline.

When a new version is better on the demo and worse on your golden tags, you do not move. The vendor's eval is not your eval. Day 16 already said this. Ops is where people forget it because the new name sounds like progress. Progress is a score on your set, not a name in a changelog.

Canary means a slice of traffic or one team, with the same eval pulse, for a named period, and a rollback pin you can actually revert at 21:00. 'Priya will try it' is not a canary. Canary without a rollback is a partial outage you chose. Who approves a model move is the same list as who waives a quality miss, because a model move can cause a quality miss. Put it on the change calendar with the golden-set result attached. A vendor email is a trigger, not an approval.

Prompt and model are a pair. Changing both on the same day is how you cannot tell what broke. Change one, run the set, then the other if you must. The change ticket names which one moved. Production prompt editing plus a silent model bump is how a week of debugging becomes folklore. Document the pin in config operators can find, not in a notebook comment. The incident page should say where the pin lives and how to revert it.

Vendors will retire the name you pinned. The email will arrive with a date. That date is the start of a project: impact, eval on the successor, cost/latency compare, canary, rollback plan, communication to users if behavior changes, change calendar, Security if the path or subprocessors change. Treat it like a platform upgrade, because it is one.

Have a written successor before you need it. When you choose a vendor on day 19, one of the five questions is deprecation plan. What is the notice period. What is the recommended successor. Can you keep the retired pin for thirty days after the date if you must. If the sales engineer cannot answer, you have a lock-in you have not priced.

Behavior change is the hidden cost. A successor that is 'better' may be more verbose, more sycophantic, worse at following your schema, more expensive. Your golden set is how you see it. Run it on the successor the week the email arrives, not the week of the deadline. If you must stay, you need a waiver and a prompt change, sequenced, not both on the deadline Friday.

Users notice tone. If the copilot starts answering like a different person, you will get tickets that are not quality fails and still hurt trust. Warn them. A one-line note in the UI for a week, and a note to champions. Change management applies to model moves. You would not swap a system of record without telling the users it looks different. Do not swap a model silently.

Budget the eval work. A deprecation is a few days of someone's time if you already have the set and the dashboard. It is a panic if you do not. This is one of the reasons day 16 exists. Ops without eval is a series of surprises with marketing names.

If you have more than one feature on the same pin, deprecation is a portfolio move. List the features, the owners, the bars, and the order you will canary. Do not move everything on one night because the email was urgent. Urgent for the vendor is not urgent for your error budget.

Have a written successor before you need it. Notice period, recommended next pin, how long the retired pin lives after the date. If the sales engineer cannot answer, you have a lock-in you have not priced. Run the golden set on the successor the week the email arrives, not the week of the deadline. A successor that is 'better' may be more verbose, more sycophantic, worse at schema, more expensive. Your set is how you see it. The vendor's eval is not your eval.

Users notice tone. If the copilot starts answering like a different person, you will get tickets that are not quality fails and still hurt trust. Warn them: a one-line UI note for a week, and a note to champions. You would not swap a system of record without saying it looks different. Do not swap a model silently. Budget the eval days so a deprecation is a project with an owner, not a Friday panic. Ops without eval is a series of surprises with marketing names.

One screen. Volume, p95 first token and complete, error rate, dollars per day and dollars per successful task, eval snapshot (recall, groundedness, must-refuse), thumbs, top refusal reasons, fallback rate. You already know a service dashboard. Add eval and money. If it is not on the screen, it is not managed.

Demand the tiles even if you cannot build them. Write them on the SLO sheet as a requirement on the engineering partner. A workstream that cannot see spend will learn spend from Finance, late. A workstream that cannot see fallback rate will not know the model has been down for two hours because the UI still showed search results. The tiles are how you learn in time to act.

Refusal reasons and fallback rate are the underrated tiles. They tell you whether policy is working or whether users are hitting a wall and leaving. A spike in refusals can be good (injection tests, out-of-scope) or bad (retrieve is empty so everything refuses). You cannot know without the tag. Build the tag. This is the same as looking at HTTP 403 versus 500. The code matters.

Do not drown. Twelve tiles on one screen, not forty. Drill-down can exist. The daily view is the twelve. You will be asked for a steering snapshot: four numbers (volume, quality, spend, latency) and one narrative. Pull them from the same screen so you are not assembling folklore the morning of the meeting.

On-call looks at this screen, not at a model playground. Who is paged for daily cap, for error rate, for eval drop. You. An engineer. Both. Write it. AI features that 'nobody really owns in ops' become orphan incidents. You already refuse to go live without an ops owner on other systems. Refuse here.

The dashboard is also how you catch silent model drift. Quality tile down, latency tile down or up, spend tile up, no change ticket in the calendar — that is a floating pin or a vendor change you did not notice. The screen makes it visible. Visibility is the control. You cannot pin what you cannot see.

Refusal reasons and fallback rate are the underrated tiles. They tell you whether policy is working or whether users are hitting a wall and leaving. A spike in refusals can be good (out of scope, injection) or bad (retrieve empty so everything refuses). You cannot know without the tag. Build the tag. This is HTTP 403 versus 500. The code matters. Put both tiles next to thumbs so you do not read a thumbs-down spike without context.

On-call looks at this screen, not at a model playground. Who is paged for daily cap, error rate, eval drop: you, an engineer, both. Write it. AI features that nobody really owns in ops become orphan incidents. You already refuse to go live without an ops owner on other systems. Refuse here. The steering snapshot is four numbers from the same screen — volume, quality, spend, latency — plus one narrative. Do not assemble folklore the morning of the meeting.