How to use a vector database

The previous post ended on a claim: once you accept that a context window is scarce, the interesting question stops being “how do I fit everything in” and becomes “how do I find the right few thousand tokens”. That is a retrieval problem, and a vector database is one of the tools for solving it.

It is also the tool people reach for reflexively, often without needing it. So this post covers both: how to use one properly, and how to tell when you shouldn’t.

From text to vectors

An embedding model takes a piece of text and returns a fixed-length vector — typically several hundred to a couple of thousand numbers. Similar text lands in similar directions in that space.

Two things to be precise about, because a lot of hand-waving lives here.

The vector does not “contain the meaning” of the text. It’s a learned compression whose geometry reflects the statistics of how language is used. That turns out to be extremely useful — it’s why “how do I cancel my subscription” can retrieve a document titled “Ending your plan”, with no shared keywords — but it also explains the failures. Embeddings are good at topical similarity and bad at precision. Two sentences that differ only by a negation, or by a version number, can sit almost on top of each other.

And an embedding is only comparable to vectors from the same model. Change the model — or even the version — and you must re-embed the entire corpus. Plan the migration path before you index a million documents.

Similarity is usually cosine similarity: the dot product of two normalised vectors, which measures the angle between them and ignores magnitude.

What a vector database is for

Given a query vector, you want the nearest vectors in a corpus. Brute force is a scan — fine for thousands of items, hopeless for millions.

A vector database gives you an approximate nearest neighbour index (HNSW and IVF being the common families) that finds almost the right neighbours in sublinear time. That word “approximate” is the deal you’re signing: you trade a small amount of recall for a large amount of speed, and most implementations expose a knob for where on that curve you sit.

It also gives you the boring things that decide whether this survives contact with production: metadata stored alongside vectors, filtering, upserts, deletes, and something to handle the fact that documents change.

RAG, minimally

Retrieval-augmented generation is three steps, and it is far less clever than the acronym suggests.

Ingest, once: split documents into chunks, embed each chunk, store the vector with its text and metadata.

Retrieve, per query: embed the query, search for the top k chunks, filter by metadata.

Generate: put those chunks in the prompt with an instruction to answer from them and to say so when they don’t contain the answer.

// Ingest
for (const doc of docs) {
  const chunks = chunk(doc.text, { maxTokens: 500, overlapTokens: 60 });
  const vectors = await embed(chunks.map((c) => c.text));

  await index.upsert(
    chunks.map((c, i) => ({
      id: `${doc.id}#${i}`,
      vector: vectors[i],
      metadata: {
        docId: doc.id,
        title: doc.title,
        section: c.heading,
        updatedAt: doc.updatedAt,
        visibility: doc.visibility, // used as a hard filter at query time
      },
      text: c.text,
    })),
  );
}

// Retrieve
const hits = await index.query({
  vector: await embedOne(question),
  topK: 20,
  filter: { visibility: "public" }, // applied during search, not after
});

// Generate
const context = hits
  .slice(0, 6)
  .map((h) => `<source id="${h.id}" title="${h.metadata.title}">\n${h.text}\n</source>`)
  .join("\n\n");

const answer = await model.complete({
  system:
    "Answer using only the sources provided. Cite the source id for each claim. " +
    "If the sources do not contain the answer, say so.",
  messages: [{ role: "user", content: `${context}\n\nQuestion: ${question}` }],
});

That is the entire pattern. Everything that makes it good or bad happens in the details below.

Chunking is where quality is won or lost

Chunking gets treated as plumbing. It is the single highest-leverage decision in the pipeline, because a chunk is simultaneously the unit you retrieve and the unit you show the model, and those two jobs want different things.

Retrieval wants small chunks: one topic per vector, so the embedding isn’t an average of five unrelated ideas. Generation wants large chunks: enough surrounding text that the answer is actually complete.

The techniques that resolve that tension:

Split on structure, not on character count. Documents have headings, sections, list items, function bodies. A splitter that cuts every 1,000 characters will cut through the middle of a sentence, orphan a code block from the paragraph explaining it, and separate a heading from the text it governs. Use the document’s own boundaries first, and fall back to size only inside them.

Carry context into the chunk. Prepend the document title and the heading path to each chunk’s text before embedding it. A chunk that reads “This is not supported on the free tier” is nearly useless in isolation; the same chunk prefixed with “Billing → Plan limits” is retrievable.

Overlap a little. A modest overlap between adjacent chunks stops an answer that straddles a boundary from being lost by both.

Retrieve small, then expand. Index fine-grained chunks for matching, but store a pointer to the parent section — and when a chunk is retrieved, send the parent to the model. You get precise matching and complete context, which is the best of both.

The right size depends on your content, and the only way to know is to measure. Which brings us to the thing most teams skip.

Evaluate retrieval separately from generation

If the right chunk isn’t in the prompt, no model will produce the right answer. If it is, most competent models will. Retrieval quality dominates model choice, and yet the reflex when a RAG system answers badly is to swap the model.

Build a small evaluation set — fifty real questions, each labelled with the chunk that should be retrieved. Measure recall@k: how often the correct chunk appears in the top k. Then tune chunking, embedding model, k, and search strategy against that number, in isolation from the LLM entirely.

Fifty labelled questions takes an afternoon and will tell you more than a month of prompt tweaking.

Metadata filtering

Store structured fields next to every vector — tenant, source, author, date, visibility, type — and filter on them.

The critical detail is when the filter runs. Pre-filtering restricts the candidate set and then searches within it. Post-filtering searches globally and then discards non-matching hits, which means asking for the top 10 and receiving 2, or 0, once the filter has run. For anything correctness-critical — tenant isolation, permissions — insist on filtering that is enforced during the search, and confirm your database does it that way. This is an access control boundary, not a relevance tweak.

Hybrid search, because vectors miss the obvious

Semantic search is bad at exactly the things keyword search is good at: exact identifiers, error codes, function names, SKUs, surnames, rare acronyms. Ask a pure vector index for ERR_CONN_5041 and you may get five chunks about connection errors in general and not the one page that names it.

So run both. Keep a lexical index (BM25) alongside the vector index, query both, and fuse the ranked lists. Reciprocal rank fusion is the standard approach because it needs no score normalisation — it only uses positions:

// score(d) = Σ over lists of 1 / (k + rank(d)); k ≈ 60 is the usual constant
function reciprocalRankFusion(lists: string[][], k = 60) {
  const scores = new Map<string, number>();
  for (const list of lists) {
    list.forEach((id, rank) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank + 1));
    });
  }
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id);
}

Hybrid search is usually the largest single quality win available after chunking, and it’s cheap to add.

The other reliable win is a reranker: retrieve a generous candidate set — say the top 50 from hybrid search — and pass each candidate through a cross-encoder that scores it against the query directly. It’s far more accurate than comparing pre-computed vectors, because it gets to look at the query and the document together, and far too slow to run over a whole corpus. Retrieve broadly, rerank precisely, send the top handful.

When a vector database is the wrong tool

Most of the time, honestly.

When the corpus is small. If everything you’d search fits comfortably in a prompt, put it in the prompt. Retrieval only earns its complexity when there’s too much to send.

When the question is structured. “What did I spend on groceries last month?” is a SELECT with a GROUP BY. Semantic similarity cannot count, sum, or compare dates. Give the model a query tool over the real database instead — and validate what comes back out of the model before you execute it. (The assistant in my Money Diary app does exactly this: tool calls are Zod-validated, and the userId comes from the session rather than from anything the model wrote.)

When lookup is exact. Ids, slugs, filenames, symbols. That’s an index, not an embedding.

When the answer must be complete. Top-k retrieval is a sample, not a set. “List every customer affected” cannot be answered by fetching the five most similar chunks.

When you already have Postgres. A vector extension in the database you already run, back up and secure is very often sufficient, and it lets you filter on real columns with real joins. Reach for dedicated infrastructure when scale or latency actually demands it, not before.

The summary worth keeping

A vector database is an approximate nearest-neighbour index with metadata. It does one thing: find text that’s topically close to other text, fast.

Everything that determines whether your system is good happens around it — how you chunk, what you filter on, whether you also search lexically, whether you rerank, and whether you ever measured recall. The model at the end is the part you should be tuning last.