Build vs buy, vendors, and the POC trap

You are buying outcomes on your eval, not a logo. The default for v1 is: buy the model API and maybe a gateway your company already approved. Build the evals, the prompt, the ingest rules, the HITL, the SLO sheet. That split keeps the IP you actually need when you change vendors, and it keeps you from a six-month platform stand-up before the first slice.

Buy a turnkey RAG or 'knowledge agent' only if it passes three tests: your ACLs, your export of prompts/evals/traces, and your golden set on your corpus (or a sanitized twin). If it fails any of the three, you are buying a demo. Demos are expensive in Security time. Passing a demo on the vendor's corpus is not one of the tests.

Build the thin slice in your own stack when you have engineers who can ship a retrieve-and-generate path in a few weeks, and when the vendor product would take longer to approve than to copy. This is common in companies that already have an enterprise model endpoint. The bottleneck is not the generator. The bottleneck is data-ready, evals, and change. Those are yours either way.

Do not build a platform. A platform is what you consider after two production slices have a pattern worth standardizing: same gateway, same eval harness, same logging shape. Starting with a platform is how a year passes and no user has a copilot. A delivery lead who delays the platform conversation is doing the company a favor. Say that in the recommendation. Someone has to.

People will argue this as identity — 'we are a build shop' or 'we are a buy shop.' Ignore identity. Write a table: what we buy, what we build, what we refuse to outsource, who owns day 91. Four rows. The argument becomes a document. Documents can be marked up. Identity cannot.

Cost at 10× belongs on the table. A vendor that looks cheap on twelve users and ruinous on one hundred and twenty is not cheap. Ask for the price at 10× volume, at 10× context, and at the next model tier. Put the numbers next to the 'we will grow this' slide they brought. If they will not quote 10×, they are selling a land-and-expand with the expand unpriced.

People will argue buy versus build as identity: we are a build shop, we are a buy shop. Ignore identity. Write four rows: what we buy, what we build, what we refuse to outsource, who owns day 91. The argument becomes a document. Documents can be marked up. Identity cannot. Your default for v1 stays boring: buy the model API and maybe a gateway already approved; build evals, prompt, ingest, HITL. That split is the IP you need when the vendor changes.

A turnkey knowledge agent that cannot export prompts, evals, and traces is a tenant you will be trapped in. Passing a demo on the vendor's corpus is not a test. The tests are your ACLs, your export, and your golden set on your corpus or a sanitized twin. Fail any of the three and you are buying a demo. Demos are expensive in Security time. Say no early. Early nos are cheaper than a SOW you unwind in quarter two.

You asked five questions on day 3. Production adds teeth. One: what interface do we own — prompts, evals, traces, indexes — and can we export them. Two: cost at 10×, in writing, including the next model tier. Three: data path in writing — region, training clause, subprocessors, retention. Four: deprecation plan — notice period, successor, how long the pin lives. Five: eval on our tasks, our golden set, our ACLs, not their corpus.

Add the operational questions that decide whether you can live with them. SSO and VPC. Log export into your SIEM, not only their UI. Support hours that match your ops, with a named severity response. Status page. A pin you control, not a floating latest. If those answers are 'on the roadmap,' they are no for a production path. Roadmaps are not controls.

Ask who the subprocessors are, and watch how fast the list appears. A vendor who cannot produce the list in the POC window will not produce it in an incident. Day 15 already made this a data-handling item. Here it is a buy decision. Unknown subprocessors on confidential text is a no. You do not need Legal to invent that rule. You need Legal to confirm it.

Ask what happens on day 91 if you stop paying. Export of prompts, evals, traces, and the index. Deletion confirmation. Whether their UI is the only place the golden set lived. If the golden set lived only in their UI, you did not have a golden set. You had a tenant. Tenants die. Your set should not.

Run the questions as a scorecard, not as a conversation you will remember. Six to ten rows, the five questions plus SSO/VPC, log export, subprocessors, support hours. Score each vendor the same week, same corpus, same set. A narrative after two demos is how the better slide wins. A scorecard is how the better path wins.

You can ask these questions without being rude. Send them ahead of the demo. Vendors who refuse to run on your set, or who need a month to answer subprocessors, have told you something. Believe them. Delivery leads who skip the email because the meeting felt good will rewrite the email after the SOW, when it is worse.

Run the questions as a scorecard the same week, same corpus, same set. Six to ten rows: the five questions plus SSO/VPC, log export, subprocessors, support hours. A narrative after two demos is how the better slide wins. A scorecard is how the better path wins. Send the scorecard ahead of the meeting so the facilitator is not surprised when you ask to run your injection row live. Surprise is how you get a reschedule and a friendlier script.

Roadmap answers are no for a production path. 'SSO on the roadmap,' 'export coming soon,' 'VPC later this year' are not controls you can operate on day 91. You may still POC if the death date is real and the data is sanitized. You may not promote on a roadmap. Write that rule down so a sponsor who loved the UI does not promote from the hallway. Controls live in the present tense. Roadmaps live in sales.

A POC without a death date becomes production by inertia. You have seen it on other tools. AI makes it faster because twelve users will build a habit in ten days and then fight you when you try to turn it off. Write the clock (ten working days), the users (twelve, named), the non-goals (no SSO to the whole company, no write-tools), the kill (the tenant dies on day 11 unless a named exec signs a production design with Security), the data (sanitized or a dedicated space), and the success bar (the eval bar, not 'people liked it').

Put the death date in the SOW and in the calendar with a meeting already booked: promote or kill. The meeting has a packet: eval on your set, data-handling note, cost at 10×, who owns day 91, Security conditions. If the packet is not ready, you kill. An incomplete packet is not a reason to extend. Extensions are how death dates die.

Named users, not 'the PMO.' Twelve humans you can write to when the tenant is going away. If the vendor wants a company-wide SSO for the POC, that is not a POC. That is a deployment. Refuse. A dedicated IdP group of twelve is a POC. Open enrolment is production with a smile.

Sanitized data or a dedicated space, not the real confidential corpus, unless Security has already signed the production path. POCs love to 'just connect Confluence' because it makes the demo sing. That connection is the data path from day 15, with copies, with subprocessors, with no handling note. You already know not to point a trial at production customer data. The wiki is production data. Treat it that way.

Success is the eval bar you brought. If the vendor's facilitator spends the two weeks collecting anecdotes, you will be asked to promote on anecdotes. Run your fifteen or forty rows on day two and on day ten. Compare. That table is the only success metric that matters. 'People liked it' can be a secondary note. It cannot be the gate.

Non-goals are what keep the POC from growing a write-path in week two because someone was excited. No tools that send. No HR spaces. No customer mail. Write them. When someone asks, you have a page. Excitement is not a change-control process.

Book the promote-or-kill meeting on day one, on the death date, with the packet listed in the invite: eval on your set, handling note, cost at 10×, day-91 owner, Security conditions. If the packet is not ready, you kill. An incomplete packet is not a reason to extend. Extensions are how death dates die. Put the death date in the SOW as well as the calendar. A calendar hold the vendor did not sign is a hope. A SOW clause is a lever.

Named users, not 'the PMO,' and no company-wide SSO for a POC. Twelve humans you can write to when the tenant is going away. Open enrolment is production with a smile. Sanitized data or a dedicated space unless Security has already signed the production path. 'Just connect Confluence' is the data path from day 15 with no handling note. You already know not to point a trial at production customer data. The wiki is production data. Treat it that way.

Demo-ware shines on the vendor's corpus, with ACLs off, on a happy-path script, with a facilitator who already knows the answers. It fades on your corpus, your ACLs, your injection rows, and your empty-retrieve cases. Your job in the demo is to make it fade early if it is going to fade. Bring three of your golden rows, including a refuse and an injection. Ask to run them live. Polite vendors will. The others have told you it is demo-ware.

Lock-in is not 'we use a vendor.' Lock-in is 'we cannot leave without losing the evals, the prompts, the traces, and the ACL mapping.' Own those four and a model swap is a project. Do not own them and a model swap is a rewrite. Day 91 ownership is the test. Write it on the scorecard as a row: export of prompts, evals, traces, index. Yes or no. Not 'API coming soon.'

Subprocessors are a data fact and a concentration risk. If your 'enterprise' copilot sends text to three unknown companies for 'safety' and 'quality,' you have three extra DPAs to think about and three extra incident surfaces. Ask for the list. Ask whether you can opt out of any of them. Ask the region. Put the answers on the handling note. If Sales has to 'get back to you,' the clock on the POC does not pause in their favor.

Facilitated success is a variant of demo-ware. The vendor's engineer sits in the tenant for two weeks and hand-tunes prompts until your twelve users smile. Then they leave. Day 91 you own a tenant nobody on your side can tune. If the POC requires their engineer to be good, the production path requires a named FTE on your side or a paid services retainer you have actually budgeted. Unbudgeted vendor heroics are not a runbook.

Slideware architectures are not designs. A reference architecture can be useful as a starting diagram. A slide that says 'ingestion, orchestration, insights, agents' is a marketing layout. Ask them to draw your data path from day 15 on their product. If they cannot put copies, subprocessors, and log retention on it, they do not have an architecture for you. They have a deck.

You are allowed to say no to a well-known logo. Brand is not a control. Plenty of good logos fail ACL, fail export, or fail your set. Plenty of boring API-only paths pass. Your recommendation should be boring if boring passes. Hiring managers for this job are not looking for a vendor crush. They are looking for a spine.

Facilitated success is demo-ware with a badge. The vendor's engineer sits in the tenant for two weeks, hand-tunes prompts until your twelve users smile, then leaves. Day 91 you own a tenant nobody on your side can tune. If the POC requires their engineer to be good, production requires a named FTE on your side or a paid retainer you have budgeted. Unbudgeted vendor heroics are not a runbook. Score operability as a row. Fail it if they will not leave the tenant alone for three days.

Ask them to draw your data path from day 15 on their product: copies, subprocessors, log retention. If they cannot, they do not have an architecture for you. They have a deck that says ingestion, orchestration, insights, agents. Reference architectures are useful as a starting diagram. Reference slideware is not a design. Bring your diagram. Make them mark it. The marks are the beginning of a handling note, or the beginning of a no.

Day 91 is the first Monday after the services team leaves, or after the POC promotes, or after the internal engineers who 'helped stand it up' rotate. Who versions the prompt. Who runs the golden set. Who is on-call for the daily cap. Who talks to Security when a subprocessor is added. Who can kill it. If those names are vendor names, you rented a demo. If they are yours, you bought a path you can keep.

Write a RACI for the production path before you promote. You already know how. Prompt owner, eval owner, corpus owner, ops owner, vendor manager. The vendor can be consulted and can run their service. They should not be accountable for your release bar. Accountability that lives off-payroll is how you get a shrug during an incident.

Knowledge transfer is a deliverable, not a vibe. Admin how-to, pin location, log export, how to add a golden row, how to rotate a key, how to disable tools. A one-hour walkthrough that was not recorded is not transfer. Put the artifacts in your wiki. If the vendor's UI is the only documentation, you will be unable to operate on the day their UI is down or their contract is in dispute.

Funding on day 91 is part of ownership. Who pays the tokens, the seats, the extra logging, the HITL hours. A POC paid from an innovation budget that expires, with no run-rate owner, is a trap. Finance should be able to see the envelope and the 10×. If they cannot, do not promote. You will be back in six weeks with an unbudgeted bill and a user community that is now a political fact.

Exit is ownership too. A written export drill: actually export traces and prompts once during the POC. If you cannot, you do not own them. Finding that out on day 91 is late. Finding it out on day 8 is the point of a POC.

When you recommend buy versus build in practice today, the last sentences are about day 91. Not about the demo. Not about the logo. About names, hours, export, and the bill. That is the recommendation a finance partner can respect and a CIO can sign.

Write a RACI before you promote. Prompt owner, eval owner, corpus owner, ops owner, vendor manager. The vendor can be consulted and can run their service. They should not be accountable for your release bar. Accountability that lives off-payroll is how you get a shrug during an incident. Knowledge transfer is artifacts in your wiki: pin location, log export, how to add a golden row, how to disable tools. A one-hour walkthrough that was not recorded is not transfer.

Funding on day 91 is part of ownership. Who pays tokens, seats, extra logging, HITL hours. A POC paid from an innovation budget that expires, with no run-rate owner, is a trap. Finance should see the envelope and the 10× before promote. Exit is ownership too: actually export traces and prompts once during the POC. If you cannot, you do not own them. Finding that out on day 8 is the point of a POC. Finding it on day 91 is late.

A platform deal before a production thin slice is a delay with a steering committee. You will spend a year on connectors, identity, 'agent frameworks,' and a backlog of use cases that have no eval. Meanwhile a department will buy a point tool, or people will keep using consumer ChatGPT. The platform will arrive to a landscape that did not wait.

Two production slices give you a pattern: the gateway, the log shape, the eval harness, the HITL UI, the handling-note template. That pattern is the start of a platform. Until then, a platform is a guess. Delivery leads are often the only people in the room who can say this without sounding like they hate architecture. Say it. Sequence it. Offer the two slices as the path to the platform, not as a rebellion against it.

Multi-use-case POCs are a cousin of the platform delay. 'Let us try it on service desk, and wiki, and sales, and code.' You will learn nothing well. One slice, one set, one death date. If they want a second slice, it is a second POC with its own clock, or it is a production design for the first. Combining them is how both stay shallow.

Executive enthusiasm is a risk item. A CEO who saw a keynote will ask why you are not buying the platform now. Your scorecard and the Atlas slice are the answer. HIPPO is tomorrow's topic. Today: do not let enthusiasm collapse the death date. Enthusiasm is not a signature from Security and it is not an eval bar.

If you are forced into a platform evaluation in parallel, contain it. Named owner (not you, if you are running the slice), named questions (the five, plus integration with the slice), named date. It does not get to reopen the slice's vendor path every week. Parallel workstreams need a boundary. You know how to write one. Write it.

Your recommendation for Atlas v1 should be boring and dated. Buy the already-approved enterprise endpoint. Build evals, prompt, ingest, HITL. No turnkey agent. No company-wide platform SOW. Revisit platform after two slices or after a written pattern, whichever is later. That paragraph is what day 21's charter will rest on. Write it so you can paste it.

A platform deal before a production thin slice is a delay with a steering committee. You will spend a year on connectors and agent frameworks while a department buys a point tool and people keep using consumer ChatGPT. Two production slices give you a pattern worth standardizing. Until then a platform is a guess. Sequence the platform as something you earn, not as a prerequisite. Offer the two slices as the path to it, not as a rebellion against architecture.

Contain a forced parallel platform evaluation. Named owner (not you if you are running the slice), named questions, named date. It does not get to reopen the slice's vendor path every week. Multi-use-case POCs are a cousin of this delay: service desk and wiki and sales and code at once, nothing learned well. One slice, one set, one death date. If they want a second, it is a second clock or a production design for the first. Combining them keeps both shallow.