Embeddings and semantic search

The embedding model reads a chunk and emits a list of numbers — hundreds to a few thousand dimensions. You do not read the numbers. The geometry is the point: chunks about the same thing sit near each other even if they share few words. 'PTO' near 'annual leave.' 'Change freeze' near 'code cutoff.' That is the trick, and it is also the failure: the geometry is meaning-ish, not identity-ish. Policy POL-1044 is a name. Names want a keyword match.

At query time you embed the question with the same model, then ask a vector index for the nearest neighbors. Those chunks are what you stuff into the generator's context. If the neighbors are wrong, the answer is wrong with a straight face. This is why 'we will fix it in the prompt' is usually a stall. The generator cannot cite a page it was never given.

Same model on both sides. If you embed the corpus with model A and the query with model B, the points are not in the same space. Distances are junk. This happens when someone 'upgrades the embedder' and re-embeds queries only, or when two vendors' vectors get concatenated because a slide said 'ensemble.' Re-embed the corpus when you change the embedding model. Budget that job. It is not free, and it is not optional.

Cosine similarity is the usual distance: how aligned are the two arrows, ignoring how long they are. You do not need the formula. You need the implication: a score of 0.82 is not '82% true.' It is 'closer than 0.61 on this model, this corpus, this week.' Do not put raw cosine scores in the UI as if they were confidence. They are useful for ranking and for a threshold you will tune on an eval set. They are not a compliance control.

Meaning-space fails in predictable ways. Short queries. Identifiers. Numbers. Negation ('not in scope for contractors'). Tables. Code. Cross-language. Near-duplicate policies from two years. The embedding will still return something nearby, because that is its job. Nearby is not 'the clause that answers this.' Keyword search, filters, and structure-aware chunking exist because meaning-space is a tool, not a mind.

Do not use a chat model as your search index. Chat models generate text. Embedding models map text to vectors. Some vendors sell both. They are different products with different evals. A frontier chat model that 'can search' is usually calling a retriever behind the scenes or pretending. Ask which, and ask for the retrieval eval.

Cost and latency sit here too. Embedding the query is a model call on every question. Embedding the corpus is a batch job when documents change. Vector indexes (pgvector, vendor DBs, specialist stores) have their own ops: sizing, filters, recall vs speed. You do not have to pick the index today. You do have to refuse a design that has no owner for 'what happens when a page is updated.'

Meaning-space is a tool you can explain in a minute and misuse for a quarter. The minute: the embedder puts paraphrases near each other; the index returns neighbors; the generator only sees those neighbors. The quarter: someone embeds the whole tenant, skips keyword search, reports 'semantic search is live,' and then cannot find POL-1044. Your standing move is hybrid plus an eval that includes identifiers, empty-gold, and a stale trap. If a vendor demo only shows paraphrases of the happy-path SOP, ask them to look up an ID and a question that should retrieve nothing. The demo will wobble. That wobble is the work. Do not apologize for making it wobble. You are not being difficult. You are refusing to fund a meaning-shaped pile.

Chunking is splitting documents into the units you retrieve. Too big: the vector is a mush of topics; you retrieve a 4,000-word blob, waste the context window, and bury the clause. Too small: you retrieve a sentence without the exception that lived in the next paragraph. Overlap (often 10–20%) is cheap insurance so a split does not cut a rule in half. These numbers are not a library default you shrug at. They are the product.

Respect document structure before you respect character counts. Headings, SOP sections, numbered clauses, ticket comments, decision-log entries — those are already units a human would retrieve. A delivery lead can demand 'chunk by heading, not by raw 500 characters' in a design review. That one sentence saves weeks of fluent nonsense. If the corpus has no structure, the first workstream is to give it some, not to embed the mess harder.

Token size is a budget. 400–800 tokens per chunk is a common starting band for policy prose, with ~80–120 tokens of overlap. Tables may want the whole table plus the heading that explains it, even if that blows the band. Code and API docs want function-level or endpoint-level units. Meeting notes want 'decision' blocks if you can detect them, otherwise they should perhaps not be in the index at all. One size is how junk gets a vector.

Overlap is not free. It duplicates text in the index, duplicates cost at embed time, and can dominate the top-k with near-copies of the same paragraph. 10–20% is a starting point, not a religion. If recall is fine and the top 5 are three copies of the same heading, you over-overlapped or you failed to de-duplicate. Measure.

Metadata is not optional: source path, heading trail, last-updated, document type, product, region, ACL / permission group. You will filter on these. 'Similar' but from the wrong country is a wrong answer. 'Similar' but from the 2019 SOP that was never archived is a wrong answer with legal flavor. Store the metadata at ingest. You cannot reconstruct ACL from the embedding later.

Drop junk at ingest or you will retrieve it forever: nav chrome, cookie banners, 'print this page,' 'click here,' duplicated sidebars, scanned PDFs with no OCR, Excel dumps with 40 empty columns. Demand a sample of 20 chunks on a slide before you fund the rest of ingest. If the sample is garbage, the index will be a garbage-shaped meaning-space. The generator will quote the cookie banner with a straight face.

Re-embed on document change, not on a nightly prayer. Event-driven when you can. A documented lag ('answers may be 24h stale') shown in the UI when you cannot. Silent staleness is an incident class of its own — tomorrow's day, but the hook is here: chunking without an update story is a one-shot demo.

Write chunk size, overlap, and structure rules as product copy, not as a library default. Example you can steal: split on H2; max 800 tokens; 100-token overlap; keep tables with their heading; drop nav and cookie banners; store path, heading trail, date, permission group; re-embed on change. Then demand a sample of 20 chunks on a slide before ingest is funded. If the sample is junk, you do not have a chunking spec — you have a tutorial. Overlap is insurance, not a religion; if the top 5 are three copies of the same heading, you over-overlapped or failed to de-duplicate. A delivery lead who will not say 'chunk by heading' in a design review will spend weeks explaining fluent nonsense. Say it. Put it in the ticket. Make changing it an eval, not a notebook experiment somebody ran on Sunday.

Pure vectors miss exact identifiers: policy POL-1044, SKU, error code, RFC number, ticket id. Keyword / BM25 search still wins on those. Hybrid means retrieve from both, merge, maybe rerank. Filters (region = EMEA, product = Atlas, audience = PM) happen as a hard constraint — before or after the similarity search, in the index — not as a hope in the prompt. 'Please only use EMEA sources' in the system prompt is not a filter. It is a wish.

Merging two lists is a product choice. Simple: take the union of top k_keyword and top k_vector, dedupe by chunk id, keep a combined score. Slightly less simple: reciprocal rank fusion, which does not require the scores to be on the same scale. Do not start by training a mixer. Start with union + filters, measure recall@k, then decide if you need a reranker.

Rerankers are a second model that scores (query, chunk) pairs more carefully on a shortlist. Use when recall is acceptable (the right chunk is in the top 20) but the top 3 are messy. Do not start there. A reranker will not find a chunk the first stage never retrieved. It also adds latency and cost on every query. Put it behind a measured need, not behind a vendor slide.

Filters are how you keep retrieval from being a clever intern with a master key. Date windows, product, region, document type, 'current vs archived,' permission group. The query 'what is the travel cap' should not retrieve a 2014 blog post because it used similar words. Default filter: not archived, in the user's ACL, in the in-scope spaces. Make the defaults visible in the UI ('searching Atlas SOPs, current only') so operators can see the cage.

When hybrid still fails: the document was never ingested, the identifier is in an image, the answer is in a table the chunker smashed, the question is about a live number in SAP. Those are not ranking problems. They are corpus, ingest, or 'this should be a tool call' problems. Retrieval eval will show them as systematic misses. Treat systematic misses as backlog with an owner, not as 'we need a better embedder.'

Search UX belongs in the design even if v1 is a chat box. Show the chunks. Let a user open the source. Let them see that a keyword hit on POL-1044 is why that chunk appeared. Transparency is how you debug and how you earn trust. A chat box that hides retrieval is a magician's hat. Magicians' hats do not pass audit.

Write hybrid as the default in the design unless an eval on your corpus says keyword-only or vector-only is enough. 'Enough' means recall@k on real questions, including the identifier questions. If your eval set has no identifier questions, you cooked the exam.

Hybrid is the default until an eval on your corpus says otherwise. Pure vector is a demo. Pure keyword misses the synonym. Filters are hard constraints in the index, not wishes in the prompt. 'Please only use EMEA sources' is not a filter. region = EMEA on the query is a filter. Rerankers come after recall is acceptable on a shortlist; they will not find a chunk the first stage never retrieved. Show the chunks in the UI so operators can see why a hit appeared. A chat box that hides retrieval is a magician's hat, and magicians' hats do not pass audit. Write the merge rule (union of top k from each side, or reciprocal rank fusion) in the design so two engineers do not invent two mixers. Start simple. Measure. Then spend.

Build 30–80 real questions with the business for a live system. For a design pack, 10 well-chosen questions is the floor, 20 is better. For each, a human marks the document and section that must be retrieved. Measure recall@k: in the top k chunks, was the needed one there? If recall@5 is 40%, stop tuning the prompt. Fix retrieve. The generator cannot save you.

This eval is yours to own as a delivery lead. Engineers will build the index. You will notice if the test questions are all happy-path from the demo script. Sit with three actual users and harvest questions from their last week of Slack, tickets, or stand-up. Include the ugly ones. Include one that should retrieve nothing. Include one whose answer lives in a table. Include an identifier. If the set is only paraphrases of 'what is our PTO policy,' you built a marketing brochure.

Precision@k is useful later (how much junk came with the gold). Start with recall. Missing the needed chunk is a hard fail for RAG. Extra junk is a cost and a distraction the generator might ignore. You will care about precision when you are stuffing 20 chunks 'to be safe' and blowing the window. First, find out whether the gold is even in the pile.

Gold labels must be specific. Not 'the handbook.' 'SOP-14.pdf, heading 4.2 Travel — contractors, 2024-03-12 version.' If two documents could answer, list both as acceptable. If the question is underspecified, fix the question. An eval row with a vague gold is how you argue about scores instead of fixing retrieve.

A 'should retrieve nothing' question is a first-class row. Gold: empty. Success: the system returns no chunk above threshold, or returns chunks the generator is instructed to treat as irrelevant, and the product refuses. Failure: it retrieves a vaguely related SOP and answers anyway. This row is how you catch the 'always returns something' disease of vector search.

Do not let the builder write all the questions. Do not let the sponsor write only the questions they will demo. Mix. Version the eval set like a document: owner, date, source of questions, known holes. When the corpus changes, some gold labels will rot. Re-judging is a job with a clock, not a hope.

Report recall@5 (and maybe recall@10) as a number with a denominator, not as 'pretty good.' 22/40 at k=5 is a conversation. 'The search feels smart' is not. Put the number on the steering slide. Put the three systematic misses next to it. That is how you get time to fix chunking instead of time to 'try GPT-next.'

Recall@k is the adult metric because the generator cannot cite a page it was never given. Report it as a fraction with a denominator, not as 'pretty good.' Gold labels are specific (file + heading + version), the question mix includes identifiers, tables, stale traps, and a should-retrieve-nothing row, and the set is harvested from real users rather than the demo script. Precision can wait until you are stuffing 20 chunks out of anxiety. First find out whether the gold is in the pile. This eval is yours. Engineers will build the index. You will notice if the questions are too kind. Sit with three actual users. The set will hurt. Good. A kind eval is how you get 90 percent on the slide and 40 percent on Monday.

k is how many chunks you retrieve. 5 is a common chat default. 10 if you will rerank down to 3–5. 20 is how you blow the context window and your bill. k is not 'more is safer.' More is noisier and more expensive. Set k from the eval: the smallest k at which recall is acceptable. Then generate from that. Stuffing 20 because recall@5 was 40% is treating a retrieval problem as a context-window problem.

Chunk size trades recall of a clause against purity of the vector. Larger chunks help when the answer needs surrounding exceptions. Smaller chunks help when documents mix topics. You find out with the eval, not with a Twitter default. Change one knob at a time. If you change size, overlap, k, and the embedder in one weekend, you will not know what helped.

Overlap trades boundary safety against duplication. If your splitter respects headings, you may need less overlap. If you split on raw tokens through unstructured dumps, you need more, and you should also ask why you are indexing unstructured dumps. Overlap will not fix a corpus that should have been edited by a knowledge owner.

Thresholds: some designs drop chunks below a similarity score. Dangerous until you have calibrated on your eval. A threshold that feels high will kill identifier questions that hybrid would have saved. If you threshold, do it after hybrid merge, and measure the empty-retrieval rate on the 'should retrieve' rows. A threshold is a recall knife.

These knobs belong in the design pack with owners and a re-tune trigger: 'we retune if recall@5 on the frozen eval drops below X, or when we change embedder, or when the corpus mix shifts (lots of tables, new space).' They do not belong as comments in a notebook.

Cost envelope: embedding the corpus is tokens × documents × recency. Query embedding is tokens × questions. Index storage is vectors × dimensions × copies (overlap). Generation cost is usually larger than retrieve cost at chat volumes, but re-embedding a giant corpus on every embedder upgrade is a real invoice. Put a back-of-envelope in the design. Day 5 math again.

When a sponsor says 'just embed everything,' your answer is this section: we will chunk by structure, store metadata and ACLs, hybrid-search, and we will not spend the ingest budget until 20 sample chunks and 10 eval questions exist. That is not process for its own sake. That is how you avoid a pretty index of junk.

Treat k, chunk size, overlap, and similarity thresholds as knobs with owners and a re-tune trigger. Change one at a time. The smallest k at which recall is acceptable is the k you generate from; stuffing 20 because recall@5 was 40 percent treats a retrieval problem as a context-window problem and will not save you. Thresholds are recall knives — do not cut until you have calibrated on the eval, and never threshold away hybrid's identifier hits. Cost sits here: corpus embed, query embed, storage of overlapped vectors, and the invoice when you change embedder and must re-embed everything. Put a back-of-envelope in the design. When a sponsor says 'just embed everything,' your answer is this section, out loud, without apology.

The practice today is not to invent a perfect wiki. It is to write 10 questions a new PM would actually ask, name the source that must come back, and define success as recall@5. You will reuse this table on day 14. Write it as if a skeptical engineer will try the questions against a real index in a week.

Harvest, do not brainstorm in a vacuum. If you have a real program wiki, skim the last month of questions in Slack, office hours, or onboarding notes. If you do not, invent a named program (Atlas is fine) with named artifacts: SOP-14, decision log, RAID, vendor contract, RACI. Specific fake is better than generic real. 'Corporate knowledge' is not a corpus.

Balance the mix using the table in c4. If you notice all 10 are how-do-we paraphrases, delete four and add an ID, a table, a should-be-empty, and a stale trap. The empty row is the one sponsors will want to drop because it 'isn't a demo.' It is the row that saves you from answering HR questions out of a project space.

Define failure as well as success. Success: gold source in top 5. Failure: gold missing, or (for the empty row) any in-scope answer generated. A separate failure: gold present but the generator ignored it — that is tomorrow's groundedness eval, not today's retrieval eval. Keep the two numbers apart. They fail independently and they have different owners.

Write two chunking rules you would demand before anyone embeds a byte. Example: split on H2; max 800 tokens with 100 overlap; keep tables whole with their heading; drop anything in /archive; store permission group. Rules beat adjectives ('chunk them reasonably').

Name the knowledge owner on the eval itself. If you cannot name who will re-judge gold labels when SOPs move, the eval will rot and you will be steering from a dead number. Rotten evals are worse than no evals because they provide cover.

Stop when the table is ugly and specific. Pretty questions are a smell. 'Explain our culture of quality' is not a retrieval question. 'Where is the definition of done for Atlas UAT' is. You are building a test set, not a keynote.

The 10-question table is a design object you will reuse on day 14, not homework. Harvest from Slack and onboarding, invent a named program if you must, and keep the mix honest. Write two chunking rules before anyone embeds a byte. Name the knowledge owner who will re-judge gold when SOPs move. Define failure separately from success: gold missing is a retrieve fail; gold present but ignored is a groundedness fail tomorrow. Do not blend them into one 'accuracy' number a vendor can hide behind. Stop when the table is ugly and specific. Pretty questions are a smell. If you would not ask it on Monday as a new PM, delete the row and write one you would.