Tokens, cost, and unit economics

Hosted models bill per million tokens, with input cheaper than output (the model spends more compute writing). A chat app that stuffs the last 40 messages plus a 30-page PDF into every turn is an input-cost problem. A verbose 'strategic' system prompt that rambles for 2,000 tokens is also an input-cost problem — and a quality problem. You already hate unused process. Treat unused tokens the same way. They are waste you pay for on every call, not a one-time document.

Rough English rule: tokens ≈ words × 1.3. A 1,000-word status pack is ~1,300 tokens in. A 400-word brief out is ~520 tokens. You do not need a tokenizer to make a go / no-go; you need an order of magnitude. Before you sign a budget, check a real tokenizer on a sample — other languages, code, and tables can run hotter. The meeting skill is the 1.3. The signing skill is a sample. If someone fights you on 1.3 in the meeting, offer to recompute on a sample after, and still use the envelope today. Do not wait for a priest.

Embedding models bill separately and cheaply per token. They are usually not the line item that kills you. The chat / completion model in a loop (agents) is. People will obsess over embedding pennies because the number is easy to see on a dashboard. Meanwhile the generator is reading eight chunks and writing a long answer eight times per question because nobody capped the agent. Look at the generator first. Look at the loop second. Look at embeddings third. If finance asks where the money went, open the hop count, not the embedding dashboard. A 6-hop graph at frontier prices will beat an embedding line item by a large integer every time.

The cost path of one question: tokenize the system prompt, the history, the retrieved chunks, and the user question (input) → run the model → emit output tokens, maybe a tool call → if a tool, run the tool, send the result back as more input, emit more output. Each hop is a billable event. A 'single question' in a demo is often one hop. A 'single question' in an agent is eight hops. Your unit is not a question. Your unit is a question × the graph.

Price cards move. The skill is not memorizing a number from a blog. The skill is: write the assumption at the top of the envelope ('$3 / million input, $15 / million output, frontier class, no cache'), compute, then swap the card when procurement sends the real one. If you cannot swap the card without rewriting the logic, you did not do an envelope. You did a screenshot. Keep the formula. Date the card. Dated cards are how you explain a miss later without looking like you guessed. Guessing is not a delivery skill.

Latency is on the same meter even though it is not dollars. Users feel anything past ~3 seconds for a copilot turn. Long context and agent loops blow that. Output tokens take wall-clock time to stream. A 1,500-token 'strategic' answer is a cost bug and a UX bug. Put p95 latency on the steering slide next to unit cost and accuracy. If you only report accuracy, someone will pick the slow expensive model and you will find out on the floor.

Prompt caching (supported by several frontier APIs) makes a large static prefix — system prompt, policy pack — much cheaper on subsequent calls. Design prompts with a stable prefix and a small tail. That is an architecture choice a delivery lead can demand. If every call reshuffles the system prompt or interpolates the date into the first line, you may break the cache. Put the volatile bits at the end. This is the same instinct as putting constants in a config and variables in the payload.

Batch APIs trade latency for a discount on offline jobs (classify 80k tickets overnight). Interactive copilots cannot batch. Do not use interactive prices for a back-office job, or batch prices for a user-facing SLA. A PMO digest that runs twice a day is a batch job wearing a copilot costume. Run it as a batch. A floor operator waiting on a lookup is interactive. Price them apart. If finance sees one blended rate, they will either panic or underfund. Split the envelope into two rows.

Agents multiply. A 'simple' agent that thinks, calls a tool, thinks, calls another tool, then answers can be 8–20 model calls per user question. Your unit is not 'a question.' It is 'a question × the graph.' Cap steps. Cap tools. Put a kill switch when daily spend exceeds X. If the builder says 'the agent will decide how many steps it needs,' you have not designed a system. You have designed a blank check. You would not give a vendor an unbounded SOW. Do not give a loop one.

Retries and repair loops are quiet multipliers. JSON failed, retry. Tool failed, retry. User hit regenerate. Each is a full bill. In the envelope, put a retry factor — 1.2 if you are honest, 2.0 if you have no eval and a flaky schema. Then go fix the schema so you can put 1.2. A delivery lead who ignores retries will be surprised by a bill that is 'unexplainably' 40 percent over. It is explainable. Look at the traces. If 30 percent of calls are a second try, you do not have a cost mystery. You have a contract bug. Structured output and a single retry cap are cheaper than arguing with finance about a vibe.

History is a quiet multiplier. Forty turns of chat, resent every time, is an input-cost problem that grows for the users who use you most — your champions. Cap history, summarize older turns, or retrieve from a session store instead of pasting the transcript. The people who love the copilot will be the most expensive if you do not cap. That is the opposite of what you want. Design for champions, not only for the first-turn demo. Put a history cap in the spec the way you put a row cap on a report. Unbounded history is an unbounded SOW the user did not know they signed. Cap it before the first champion hits turn 40.

Caching is not magic memory. It does not mean the model 'knows' the policy pack between users unless the vendor's cache is actually shared on that prefix, which has rules. Read the product doc. Assume cache is a discount on a repeated prefix for a window of minutes, not a knowledge base. Still demand it for production prompts with a large stable pack. The discount is often large enough to change the envelope. Leaving it off is leaving money on a table you will later RAID.

Copilot: 200 users × 6 queries/day × 22 working days. Each query: 4,000 input tokens (prompt + a bit of history) + 500 output. Monthly input 200×6×22×4,000 = 105.6M tokens; output 13.2M. At a notional $3 / $15 per million in/out, that is ~$317 in + $198 out ≈ $515/month in model spend — before retries, tools, and the 10× you should fear. Write the assumption at the top. This number is a teaching card, not a quote. Plug in the real card before you brief a VP.

If you stuffed 40k tokens of SharePoint each turn instead of 4k, multiply input by 10: ~$3,170 input plus the same output, ~$3.4k/month at this volume. Still not a company-ending number — until you 10× the users, or add an agent, or both. 2,000 users at 40k input is ~$31k/month input at this card, before output, retries, and tools. That is a budget line. That is also a latency line. The stuffing is doing double damage. Retrieval is a cost control, not only a quality control.

Agent version of the same copilot: 6 hops per question, same sizes each hop (this is pessimistic but useful). Calls go from 200×6×22 = 26,400 to 158,400. Input tokens ~634M, output ~79M. At $3/$15: ~$1,900 in + $1,185 out ≈ $3.1k/month at 200 users. At 2,000 users: ~$31k. At 2,000 users plus stuffing: ugly. Cap hops at 3. Cache the prefix. Retrieve 2–4 chunks not 20. Those three design changes are how a delivery lead cuts a 5× bill without 'using a worse model.'

Per-user unit. $515 / 200 users ≈ $2.60 per user per month at the thin copilot. That is coffee. Finance will not care — until you add seats, hops, and stuffing. Report unit and monthly and 10×, always. A unit of $2.60 looks safe. A unit of $26 with an agent and stuffing, times 2,000, does not. The 10× is not theatre. It is the first year of a successful copilot. Successful is the stress case. Failed POCs are cheap. That is why cheap POCs do not prove anything about cost.

What would cut input in half on this copilot. (1) Stable system prompt + cache. (2) History cap at 4 turns or a summary. (3) Retrieve 3 chunks of 400 tokens instead of a 12k paste. (4) Route 'what's the status of ticket X' to a tool lookup instead of a long generate. (5) Stop asking for a 800-word brief when a 120-word digest would do — that one is output, but it also reduces the next turn's history. Pick one. Put it on the slide. A slide with a number and no design change is a complaint. A slide with a number and a change is a decision.

Latency envelope, same copilot. One hop, 4k input, 500 output: often inside 3 seconds on a frontier API, not always. Six hops: you are in 10–20 seconds if you are lucky. Floor users will not wait. Steering-pack generation overnight can wait. Do not use one SLA. Interactive vs batch, again. If the builder wants an agent on the interactive path, they owe you a p95 under 3 seconds or a UI that does not pretend it is chat. A spinner for 20 seconds is a failed interface, even if the answer is good.

RAG over 50k chunks, 3,000 questions/day. Retrieval is cheap; the generator still sees 3–8 chunks. Cost lives in the generator and in re-embedding when docs change. Budget a re-index job, not just Q&A. People see '50k documents' and multiply 50k by the generator price. That is the stuffing error in a different costume. The index can hold 50k. The prompt should hold 3–8. If your prompt holds 50k, you did not build RAG. You built a paste with extra steps.

Embeddings math, order of magnitude. 50k chunks × 400 tokens = 20M embedding tokens to build the index. At a notional $0.10 / million, that is $2. Noise. A full re-embed after a corpus cleanup is still noise at this size. It stops being noise at tens of millions of chunks or at a vendor who prices embeddings like generation. Still budget the job: who runs it, how long it takes, what happens to Q&A while it runs. Ops cost is not token cost, but it is a RAID line ('index stale for 14 hours').

Per-question generator: 8 chunks × 400 tokens = 3,200 tokens of evidence, plus 600 of system, plus 200 of question, plus 400 of history ≈ 4,400 input; 400 output. At $3/$15: about $0.013 input + $0.006 output ≈ $0.02 per question. 3,000/day × 22 = 66,000 questions ≈ $1,320/month. That is the honest RAG envelope at this card and this retrieve size. If someone quotes you $40k/month for this shape, they are stuffing, looping, or using a price card from a different planet. Ask which.

What blows RAG cost. Retrieving 20 chunks 'to be safe' (input × 2.5). Rewriting the query with a frontier model, then retrieving, then generating, then critiquing with a second model (4× hops). Re-ranking with a second model on every call when a first-stage retrieve was enough. Users who paste the previous novel into the next question. Multi-hop 'research agents' over an internal wiki. Each of these can be right for a hard task. None of them should be the default for 'what is the password-reset SOP.' Route the hard 5 percent. Do not pay research-agent prices for SOP lookup.

Corpus change is the RAG-specific tax. Policy pack updates every Thursday. Do you re-chunk and re-embed the changed docs (good) or the whole corpus (lazy, still cheap at 50k, not cheap at 50M)? Do you version the chunks so citations point at the Thursday pack, not last month's? A wrong citation is a quality bug that also wastes the next generator call when users retry. Put 'index freshness' on the RAID log next to token spend. Stale index plus a fluent answer is how you ship last quarter's policy with this quarter's confidence.

10× the questions: 30k/day. Generator 10× ≈ $13k/month at this card. Index does not 10×. This is why RAG at scale is usually a generator problem, not an embedding problem. Design changes that hold the 10×: smaller chunks but not more of them, cache the system+tooling prefix, a small model to refuse off-corpus questions before the frontier generator runs, batch the overnight digest instead of 3,000 interactive calls. Same story as the copilot. Different object. Same adult. Write the axis on the envelope: 10× questions, not 10× documents. Multiplying the corpus is the stuffing error in a forecast costume.

A POC is a cost liar for structural reasons. Users are few and polite. Prompts are short because the corpus is a folder of twelve PDFs. Nobody built the agent yet. Nobody left history on. The model is on a credit. Finance is not in the room. Then the POC 'works' and someone multiplies seats by 50 without multiplying tokens per seat. The invoice is how you meet finance. You would rather meet them with an envelope. Put the envelope in the charter on day one of the POC, not the week the invoice lands. An envelope that was always there is a forecast. An envelope invented after the bill is an excuse.

How to 10× without theatre. Take the POC trace of a typical question. Count input tokens, output tokens, hops. Multiply questions × users × working days × 10. Add a retry factor. Add a champion-history factor (1.5× input if you have no cap). That is the stress. If that number needs a meeting, you needed the meeting before you promised the roll-out. If that number is coffee, say so, and still cap the hops so it stays coffee. Coffee with an unbounded graph becomes a budget line the first month champions actually use it. The cap is how coffee stays coffee. Write the cap as a number of hops, not as a hope.

What 10× usually finds. The retrieve is a paste. The agent is unbounded. The system prompt is a novella. The batch job is priced as interactive. The embedding bill is fine and everyone was staring at it. The interactive SLA is attached to a 12-hop graph. Fix those before you negotiate a discount. A discount on a stupid graph is a cheaper stupid graph. Vendors will happily sell you reserved capacity for it. Do not buy a reserved stupid graph.

The political use of 10×. Sponsors hear 'cheap' in the POC and 'expensive' at roll-out and conclude you sandbagged. If the steering pack on day one of the POC had the 10× line, you did not sandbag. You forecasted. Put the line in the charter next to the kill criteria. 'If 10× monthly model spend exceeds $X with the designed graph, we simplify the graph or we do not roll out.' That is a kill criteria a finance partner can love. It is also how you stop a transformation office from promising 80 copilots on a POC price.

Worked contrast. POC: 12 users, 20 questions/day, 2k in / 300 out, 1 hop, 20 working days. Input 9.6M, output 1.44M → ~$29 + $22 ≈ $51/month. Someone says 'see, it's free.' 10× users and a real retrieve (8k in) and 3 hops: 120 users, 20 q/day, 8k in, 300 out, 3 hops, 22 days. Calls 158.4k. Input 1.27B, output 47.5M → ~$3.8k + $713 ≈ $4.5k/month. Still maybe fine. 10× again to 1,200 users: $45k. The lie was not the math. The lie was treating $51 as the math.

Always say which 10×. 10× users, 10× questions per user, 10× hops, 10× input size are four different stresses. Do the one that matches how the program would actually grow. A successful internal copilot grows users first. A public-facing assistant grows questions first. An agent program grows hops first, by accident. Name the axis. A generic '10×' that nobody can audit is just a bigger vibe. Finance can challenge a named axis. They cannot challenge a vibe, so they will invent a worse number for you. Name it first.

Cost risk: volume assumptions wrong, agent loops unbounded, model price change, no cache, stuffing dressed up as RAG, retries on a broken schema, champion history uncapped. Mitigation: cap tools/steps, cache prefixes, route easy traffic to a small model, kill switch when daily spend exceeds X, history cap, retrieve-n not retrieve-all. You already know how to write this. The new part is the unit: tokens × calls × price, with a 10× stress. Bring that slide. A RAID item that says 'AI is expensive' cannot be closed. A RAID item that says 'if weekly spend > $Y, cap hops to 2' can. Closable items get closed. Bring the closable one.

Assign owners like any other RAID item. Spend kill switch: platform owner. Graph cap: the FDE or engineer who owns the loop. Price-card watch: the person who signed the vendor (it will change). Eval that a cheaper model still holds: AI PM or you. If all four owners are 'the AI team,' you have no owners. Split them. Cost is not a technicality that will be cleaned up after go-live. Cost is a go-live criterion. Write it next to the eval bar and the data path. A go-live that cannot name the spend owner is a go-live that will page you on a Sunday when the bill doubles. Name the owner before the cutover, not after the invoice.

Leading indicators, not only the invoice. Daily token burn vs the envelope. p95 hops per question. Cache hit rate. Percent of calls that hit the expensive model vs the cheap router. Retry rate. If retry rate is 30 percent, you do not have a cost mystery. You have a schema or a retrieve bug. Put two of these on the weekly pack. You already put defect aging on the weekly pack. This is defect aging for money. Pick burn-vs-envelope and p95 hops if you can only have two. They catch stuffing, unbounded graphs, and silent model swaps before finance does. Finance finding it first is a process failure, not a surprise.

Price change is a real RAID, not a joke. Vendors drop prices and they also retire cheap model names. A deprecation can move you to a more expensive successor. Re-run the envelope on the successor before you accept the auto-upgrade. Sometimes the move is 'stay on the old name until it dies, then re-eval.' Sometimes it is 'the successor is cheaper and better; take it after the eval.' Never 'it upgraded itself over the weekend and the bill doubled.' That sentence is a change-control failure.

What not to put on the RAID log: 'AI is expensive' as a vibe. That item cannot be closed. Put a number, an assumption, a trigger, an owner. 'If weekly spend > $Y for 2 weeks, cap hops to 2 and page the owner.' Closable. You look like a delivery manager. 'We should watch costs' looks like a newsletter. Closeable items get closed. Newsletter items get copied into the next pack. You already hate un-closeable RAID. Do not add one with the word AI on it.

Tie it to the seats. Transformation leads who cannot 10× a copilot will fill a portfolio with 80 un-enveloped ideas. Solutions engineers who skip the envelope will sell a graph the customer cannot afford. FDEs who skip it will be surprised in the customer's cloud bill. Delivery leads who bring the envelope will be invited back. This is a boring superpower. Boring superpowers ship. If your seat is transformation, the envelope is how you say no to use case 81. If your seat is solutions, it is how you survive year two of the contract.

Finance will not wait for a perfect model. They will wait for an envelope with an assumption line. Tomorrow's number is allowed to be ugly if it is labeled. 'Placeholder card $3 / $15 per million, 80 users, 10 queries, 8k / 600, 22 days, one hop, retry 1.2, no cache. Monthly about $X. 10× users about $10X. If we add a 6-hop agent, multiply calls by 6.' That paragraph is a deliverable. 'We are still figuring out AI costs' is not. You can write the paragraph in twenty minutes from day 5's method. Write it.

Separate the number they asked for from the number they need. They asked for 'what does Copilot cost.' They need: model spend at expected volume, model spend at 10×, the graph assumption (hops), and whether this is interactive or batch. If you give them a single monthly figure with no graph, they will treat it as a commitment and you will miss it when hops appear. Put the graph in the sentence. 'At one hop, coffee. At six hops, a budget line.' Finance can work with a range that has causes. They cannot work with a vibe that later doubles.

Do not hide the placeholder card. If you used today's teaching numbers, say so, and offer to swap the card when procurement sends the real one. Finance would rather see a formula than a screenshot of a vendor blog. The formula is tokens × calls × price × retry. Calls is users × questions × hops × days. Write it once at the top of the sheet. Then every argument is about an assumption, which is a decision, not about whether you 'know AI finance.' You know envelopes. That is the job.

What to refuse. A single 'per user per month' that assumes a consumer-seat analogy. Seats are not tokens. A user who asks two questions with a 2k prompt is not a user who asks twenty with a 40k paste. Per-user is a derived metric after you have the graph, not a starting price. If they need a per-user for a business case, give it with the graph attached: '$2.60 at the thin copilot, $26 at agent-plus-stuffing, for this volume.' Derived, labeled, killable. A naked per-user will be copied into a 5,000-seat fantasy.

Bring one design change with the number, or the meeting becomes a request for a discount. 'We can cut input in half by retrieving three chunks and caching the prefix. That takes a week of FDE time. Without it, 10× is $Y. With it, $Y/2.' Now finance has a choice that is not 'kill the program' or 'pay whatever.' You already run this move on vendor SOWs. Run it on the graph. The graph is the SOW the model signed with you. Cap it.

After the meeting, put the envelope on the RAID log with a trigger. If weekly spend exceeds the envelope by 20 percent for two weeks, the owner caps hops and pages you. Finance asked for a number tomorrow because they have been burned by unmetered tools. Give them a number and a trigger. Then they will stop asking every week. A number without a trigger is a promise. A number with a trigger is operations. You are here for operations.