Calling an LLM API like a grown-up

POST to an HTTPS endpoint. Header: Authorization: Bearer <key>. Body: model, messages[], optional temperature, max_tokens, tools, response_format. Response: a message plus usage (token counts). That is the whole trick. There is no hidden ritual. There is no 'the model already knows our tenant.' Each call is a self-contained HTTP request. If the body does not contain the documents, the model does not have the documents.

The URL is a product decision pretending to be infrastructure. Azure OpenAI, Bedrock, Vertex, Anthropic, a company gateway — they all wrap the same idea with different path shapes and auth. Your design should name the interface you own (messages, tools, prompt version) and the door you use (the gateway). Do not tattoo a vendor path into every workstream. Doors change. The messages array should not have to.

messages[] is the conversation the model is allowed to see: system (the spec), optional developer instructions, then user and assistant turns, plus tool results if you are in a loop. Order matters. A system prompt buried after three user dumps will be treated as a suggestion. Put the contract first. Put the instance (this question, this retrieved packet) in the user turn. You already split brief from evidence in human work. Do it here.

temperature near 0 makes outputs more repeatable — what you want for extraction, classification, RAG answers you will eval, and anything a regulator might reread. Higher temperature is for brainstorming and for 'give me three phrasings.' It is not for RAID logs, not for policy answers, not for the production default. If a vendor demo runs at 0.7 because it 'feels alive,' that is a demo setting. Write temperature 0 (or 0.1) in the spec and make changing it a change request, not a constant someone flips on Friday.

max_tokens is a cost cap and a runaway cap. It is not a quality dial. Too low, the model stops mid-sentence and you ship a truncated JSON. Too high, a verbose model writes an essay you pay for. Set it from the output contract: a 150-word digest does not need 4,000 tokens of room. For JSON extraction, set it from the schema size plus slack. Log completion_tokens so you can see if you are routinely bumping the cap — that is a prompt problem or a stuffing problem, not a reason to 'buy more tokens' blindly.

response_format and JSON mode are how you demand structure. They reduce, they do not eliminate, malformed output. The client still parses. On parse failure you retry once with a 'return valid JSON only' repair prompt, then you fail visibly. Silent fallback to prose is how extraction pipelines rot. If the task is prose, do not ask for JSON. If the task is structured, do not accept prose.

The response you actually use is nested. Content sits under choices or under a content array depending on the vendor. usage sits beside it: prompt_tokens, completion_tokens, sometimes cached tokens. Log usage on every call. It is the only honest cost signal you will have until finance builds a better one. If usage is missing, your client is incomplete.

Walk a real request with a sponsor once. Whiteboard the URL, the bearer header (the key is not written down), the body fields, and the usage object coming back. Then ask which of those they thought 'the platform' handled invisibly. Most of the room believes the model remembers last week's file. Showing that every call is a self-contained POST is the cheapest way to kill a chatbot-of-everything. Put the drawn request in the design pack. When someone later wants to 'just add memory,' you point at the picture and ask what will be stored, where, with whose ACL, and on what retention clock. That conversation is the job. The JSON is how you start it. If you cannot draw this request without looking, you cannot review a vendor's 'simple integration' slide, and that slide is how programs lose six months.

Streaming sends tokens as they are born so the UI can type. Users forgive an 8-second answer that starts in 400ms. They do not forgive a spinner that sits there for 8 seconds and then dumps a paragraph. Delivery implication: streaming is a requirement for interactive copilots. It is optional for overnight jobs, batch extraction, and eval runs — those should be one-shot, logged, easy to retry as a unit.

Streaming does not change the model. It changes when bytes arrive. The full answer is still the full answer. You still pay for the tokens. You still have to parse JSON at the end if you asked for JSON (you cannot trust a partial object). If your product needs structured output, stream for the human-readable summary, or wait for the complete JSON. Do not try to parse half an object into a ticket body.

You still need a timeout. Streams can hang with an open connection and no tokens. A watchdog that cancels at 30 seconds (or whatever the UX budget is) is not pessimism. It is the same control you put on a report job. On cancel: fail the user visibly, log the partial, do not retry forever. If the action was a write, retry only with the same idempotency key.

Partial output is a product question. Do you show the tokens as they arrive (copilot chat), or do you buffer until a sentence boundary, or until a citation block is complete? Showing a claim before its citation arrives trains users to trust unfinished answers. For policy copilots, buffer the first complete cited paragraph, then stream the rest. For brainstorming, stream immediately. Write the choice down. Do not leave it to the frontend engineer at 5pm.

Cancellation is part of streaming. Users hit stop. That must abort the HTTP request, stop billing if the vendor supports it, and leave a trace that says 'user cancelled' rather than 'error.' If a write-tool was about to fire, stop is a hard no. The confirm UI should not be reachable from a cancelled stream.

Eval and replay hate streams. For anything you will score, store the final assembled text, not the token events. Keep streaming in the UX path. Keep one-shot in the eval path. Same prompt version, same model, same temperature. If eval cannot reproduce what the user saw, you do not have eval — you have a vibe check.

Cost and caching interact with streaming less than people think. Prompt caching, if the vendor offers it, is about the prefix of the prompt (system + tools + static packet), not about whether you stream. Your job is to keep the static prefix stable so cache hits exist. Randomly shuffling the system prompt or injecting timestamps into it is how you pay full price every call.

Write streaming into the UX spec as three bullets, not as 'we'll stream.' One: interactive copilots stream; eval and batch do not. Two: policy-shaped answers buffer until the first citation exists, then stream. Three: stop/cancel aborts the HTTP request, records 'user cancelled,' and cannot reach a write. Those three bullets prevent a frontend engineer from inventing a fourth behavior at 5pm. They also give you something to demo that is not a model trick. Sponsors feel streaming. They do not feel temperature. Use that. Then keep temperature at 0 anyway, because the thing they feel is not the thing you eval. If the team cannot reproduce a streamed answer in the eval path, you do not have eval — you have a stage show with a bill attached.

429 rate limit — the vendor is throttling this key or this tenant. Back off with jitter, queue if you are a batch, tell an interactive user 'busy, retry in a moment.' Do not hammer. Do not spin up more workers. A 429 storm is how POCs get their keys paused and how production spends a morning in a vendor ticket. Put the rate limit in the design as a number (requests per minute) and as a user sentence.

5xx — their problem, your user. Retry with jitter a small number of times (2–3), then fail visibly. Infinite retry is how a lunchtime outage becomes an overnight cost spike when the vendor returns. Distinguish 500 (retry) from 503 with Retry-After (wait that long) from 529 or overloaded (shed load, degrade). Your runbook can be a table. It should not be 'the engineer will know.'

Timeout — nothing came back in the budget. Causes: the model is slow, you stuffed too much context, the network is sad, a stream hung. Mitigations in order: shorter prompts, fewer chunks, a cheaper/faster model for a first pass, a higher timeout for batch than for chat. A 120-second timeout on a copilot is not generosity. It is a frozen page. Chat: 20–40s. Batch: longer, but still capped.

Content filter / safety refusal — a legitimate outcome. Do not silently retry around safety with a rephrased prompt. Show a sentence: 'this request was blocked by the safety policy; rephrase or talk to a human.' Log the code, not the full blocked text if that text is itself the problem. Sponsors who want you to 'turn off the filter' are asking for a policy exception. Treat it as one: named, time-bounded, with an owner, not a config flip in the client.

Empty or garbage content — HTTP 200, choices[0].message.content is '' or a lone space, or the model returned a refusal in prose that your parser did not expect. This is the silent failure that trains users to distrust the product. Map it to a sentence and a metric (empty_response_rate). Retry once only if you have reason (transient). Otherwise fail. A UI that shows a blank bubble is a defect, not a edge case.

Context overflow — 400, maximum context length. You stuffed too much. That is a retrieval bug or a history bug, not a 'need a bigger model' ticket by default. Trim history, retrieve fewer chunks, summarize the packet, or raise the overflow as a product incident against the chunking spec. Buying a 1M-context model to avoid retrieval discipline is how you get a slow, expensive, still-wrong answer.

Retries must be idempotent for write-tools. Retrying 'create ticket' without a key creates three tickets. You have seen this in integration projects. Same muscle. Every write carries an idempotency key you generate before the first attempt and reuse. Reads can retry more freely, but even reads can hurt if they trigger downstream billing or lock a record. When in doubt, the client spec says which endpoints are retry-safe.

Put the failure catalog on one slide: timeout, 429, empty 200, unsafe/filtered, 5xx, context overflow, plus write-duplication if any tool can mutate. Each row: user sentence, retry yes/no, metric, owner. If a row is missing, the UI will invent a blank bubble and you will debug from screenshots. This is the same table you have used on integrations for years. The only new rows are empty 200 and content-filter. Empty 200 is the one teams skip because HTTP said OK. Content-filter is the one sponsors want to 'turn off.' Both belong in RAID. Both belong in the client spec. If you cannot fill the table in fifteen minutes, you do not understand the product yet — you understand the happy path, which is the part that does not need you.

You want: request id, model, token usage, latency, tool names, eval scores, a hash or id of the prompt version. You often should not want: raw user text in a shared bucket if it can contain PII, secrets, health, HR, or customer-identifying content. Redact, sample, or store in a system with access control and a retention clock. 'We log everything so we can debug' is how you accidentally build a second CRM with worse ACLs.

Prompt logs win lawsuits and lose them. They prove what the system said, which prompt version, which retrieved chunk. They also store the user's pasted payroll data. Put Legal on the design once: what is stored, where, who can query, retention (30 days vs 1 year vs 'until the program dies'), and whether logs are in scope for a DSAR. Do not wait for the incident. The incident is a poor time to discover you have been retaining everything.

Redaction is a spec, not a hope. List the patterns: API keys, bearer tokens, email addresses, account numbers, national ids if they appear in your domain. Run redaction before the log leaves the box. Test it with a hostile input that contains a key. If the test log still shows the key, you do not have redaction. Sampling (log 1 in N full payloads, always log metadata) is acceptable for high volume if the sample is still access-controlled.

Access is the part delivery managers forget. Who can grep prompts? The whole engineering Slack? A restricted group? Support? A vendor success manager? Write the list. If the answer is 'anyone with cloud-console reader,' you have a problem. Prompt logs are closer to email content than to CPU metrics. Treat them that way in the design even if the first sprint stores them in the same place.

Retention needs a clock and a name. 'Indefinite because we might want evals later' is not a policy. Keep a separate, minimized eval corpus (questions you wrote, redacted answers you approved) rather than turning production logs into an eval set without consent. Production traces for debug: short. Eval sets: curated, owned, no secrets.

What you log for cost: model, prompt_tokens, completion_tokens, cached tokens if any, tool-call counts, retries. Roll these up per feature flag and per tenant. The first real finance question will be 'which copilot is the bill.' If you cannot answer from logs, you will answer from panic.

Incident review uses the trace. request id from the UI, through the gateway, to the vendor. If you cannot join those, you cannot say whether the bad answer was retrieval, prompt, model, or a timeout the UI swallowed. Putting request id on the user-visible 'something went wrong' footer is a product requirement, not an engineering nicety. Users will screenshot it. That is what you want.

Bring Legal a one-pager, not a philosophy. Fields kept always: request id, model, prompt version, usage, latency, tool names, status. Fields restricted: user text and retrieved chunks, redacted, short retention, named accessors. Fields never: keys, cookies, auth headers. Retention clock: debug traces vs curated eval corpus, different numbers. DSAR: are prompt logs in scope. If Legal cannot answer DSAR from your page, you will answer it from a panic. Do this once per program, not per feature. Features inherit the policy. A copilot that 'just logs to the shared bucket because the last app did' is how you build a second CRM with worse ACLs and no owner. You already knew not to put passwords in the RAID log. Prompt logs are closer to email than to CPU. Treat them that way in writing.

A production client is not 'the HTTP library.' It is a set of controls: timeout, retry policy, backoff, idempotency, a circuit breaker or kill switch, and a queue if you are batch. You have shipped integrations. This is the same object with a model on the other end. Write the controls in the design so they are not invented during an outage.

Timeouts come in layers. Connect timeout (give up on the TCP handshake). Overall request timeout (give up on the answer). UX timeout (what the user waits). They are not the same number. A batch job can wait 120 seconds. A copilot cannot. If the vendor p99 is 8 seconds and you set 5, you will fail healthy calls. Measure. Then set. Then put the number in the spec.

Retry policy: which status codes, how many times, with exponential backoff and jitter. Jitter matters. Twenty workers retrying on the same clock is a thundering herd. Cap total retries. Retry 429 and 5xx. Do not retry 400 (your body is wrong), 401 (your key is wrong), 403 (your authz is wrong), or a content-filter refusal. Retrying those is how you look like an attacker or a fool.

Idempotency is the rule that retrying the same action does not create a second effect. For chat reads, the 'effect' is a bill and a log line — usually acceptable. For tools that write (create ticket, send email, refund), the effect is the business object. The client generates a key before the first attempt, sends it, and reuses it. The server stores the result of that key. This is not optional on write paths. It is the difference between a retry and an incident.

Kill switch: a flag that stops sending traffic to the model without a deploy. Error rate, spend, or a vendor incident should trip it. The UI degrades to 'the copilot is unavailable; here is search / a form / a human queue.' If your only kill is 'we will undeploy,' you will hesitate, and the bill will not. Delivery owns the trip criteria. Engineering owns the flag. Product owns the degraded UX sentence.

Queues turn bursts into bills you can predict. Interactive chat should not sit behind a 20-minute queue. Batch extraction should. Mixing them on one key is how the CEO's demo dies at 10am because overnight jobs ate the rate limit. Separate keys or separate queues per class of traffic. This is the same as isolating batch from OLTP. You already believe in that. Apply it.

Put the whole thing in a table in the spec: error class, retry?, user sentence, metric, owner. That table is the practice today. If an engineer can implement from it without a Slack thread, you did the job. If they have to ask 'what about timeouts,' you wrote a vibe.

Idempotency is the control interviewers use to see if you have shipped an integration. Say it plainly: the runtime mints the key before the first write attempt and reuses it on retry; the model does not mint it; reads may retry more freely; 400/401/403/filter are not retried. Then name the kill switch: a flag, trip criteria (error rate, spend, vendor incident), degraded UX sentence, who can flip it. If those two paragraphs are in the spec, an engineer can build a grown-up client without you. If they are only in your head, you will be on Slack all afternoon, which is the failure mode this practice exists to prevent. Write them. Paste them into the ticket. Walk away on purpose. A delivery manager who cannot leave the room has not specified the client; they have become the client.

Defaults are product. temperature=0, max_tokens sized to the output contract, timeout sized to the UX, stream on for chat and off for eval, model name pinned to a dated version not to 'latest.' 'Latest' is a surprise waiting to happen. Vendors move quality, verbosity, and refusal behavior under your feet. Pin, eval, then upgrade on purpose.

Temperature is not 'creativity.' It is sampling randomness. At 0 the model still is not a database — it can still be wrong — but it is more stable, which is what evals need. If two runs of the same extraction prompt at temperature 0 disagree, you have a task that is too loose or a model that is too weak, not a reason to raise temperature. Raising it to 'get a better answer' is how you make the eval set a lottery.

max_tokens vs the context window: two different budgets. The window is input plus output. max_tokens is only the output cap. People confuse them and then wonder why a 128k model still 400s. Because they retrieved 120k of chunks and asked for 8k of answer. Retrieval discipline is the fix. A bigger window is a last resort and a cost decision.

Top-p, frequency penalty, seed, logit bias — leave them alone unless an engineer has a measured reason. You can run a whole program on temperature, max_tokens, and a decent prompt. Every extra knob is a place a well-meaning person will turn at 6pm and not tell you. Freeze the knobs in the spec. Changes go through the same eval as a prompt change.

Model aliases are a trap. gpt-x-latest, claude-latest, 'whatever Bedrock has as default' will move. Use the explicit version your eval was run against. Schedule a re-eval when you bump. That schedule is a RAID item: 'model deprecation on 90 days' notice, owner, eval set ready.' You covered deprecation as a vendor question in week 1. Here it becomes a line in the client spec.

Cost controls belong next to these defaults: max spend per tenant per day, max tokens per request, a cheaper model for the first pass or for classification, cache the static prefix. Day 5 math applies. If you cannot estimate cost at expected volume and at 10×, you do not have a production default. You have a hope that finance will not notice.

Write the production defaults in one block in the spec so they can be pasted into config: model pinned, temperature 0, max_tokens N, timeout T, stream yes/no, retries 2 on 429/5xx with jitter, no retry on 4xx except 429, idempotency on writes, kill switch named. That block is more valuable than a paragraph about 'we will use the API responsibly.'

Freeze the production defaults in one config-shaped block so they can be copied: model pinned to a dated name, temperature 0, max_tokens sized to the output contract, timeout sized to the UX, stream on for chat and off for eval, retries 2 on 429/5xx with jitter, no retry on other 4xx, idempotency on writes, kill switch named, prompt version required. Changing a default is a change request with an eval, not a Friday constant flip. 'Latest' is banned. Top-p and penalty knobs stay untouched unless someone has a measured reason. Cost at expected volume and at 10x sits next to this block, or you do not have production defaults — you have a credit card and a hope that finance will not notice until quarter-end. Pin, measure, upgrade on purpose. That sentence is the whole vendor relationship for this layer.