RAG from First Principles

A language model only knows what was in its training data, frozen at some cutoff date. It has never seen your company's documents, last week's news, or the ticket a customer opened this morning. Retrieval-Augmented Generation (RAG) is the technique that lets a model answer questions about knowledge it was never trained on, by fetching relevant text at query time and placing it in the prompt.

The idea is simple: do not make the model memorize your knowledge; let it read the relevant page just before it answers, the way a person looks something up.

The Pipeline

Ingest (once):
  documents -> split into chunks -> embed each chunk -> store vectors

Query (every question):
  question -> embed -> find nearest chunks -> stuff into prompt -> generate

Step 1: Chunking

You cannot embed a 100-page document as one vector and expect useful retrieval. You split it into chunks - typically a few hundred tokens each. This is the step that quietly determines whether your whole system works.

Chunk too large and each vector becomes a blurry average of many topics, so retrieval returns "sort of related" text. Chunk too small and you sever the context a sentence needs to make sense. Two rules that help: split on natural boundaries (paragraphs, headings, sentences) rather than a fixed character count that cuts words in half, and use a small overlap between consecutive chunks so a fact that straddles a boundary is not lost.

Step 2: Embeddings

An embedding model turns a piece of text into a vector - a list of numbers - such that texts with similar meaning land near each other in that space. "How do I reset my password?" and "I forgot my login credentials" produce nearby vectors even though they share almost no words. This is why RAG retrieves on meaning, not keywords.

embed("cancel my subscription")  -> [0.021, -0.44, 0.13, ...]  (e.g. 1536 numbers)
embed("how do I stop being billed") -> [0.019, -0.41, 0.15, ...]  (very close)

Step 3: Storage and Retrieval

You store every chunk's vector in a vector database (or, for small corpora, just an in-memory array). At query time you embed the question and find the chunks whose vectors are closest, usually by cosine similarity. Those top-k chunks are your retrieved context.

def retrieve(question, index, k=5):
    q = embed(question)
    scored = [(cosine(q, chunk.vec), chunk) for chunk in index]
    scored.sort(reverse=True)
    return [chunk for _, chunk in scored[:k]]

Step 4: Generation

Finally you build a prompt that places the retrieved chunks in front of the question and instructs the model to answer from them:

Answer the question using only the context below. If the answer is
not in the context, say you do not know - do not guess.

Context:
{retrieved_chunk_1}
{retrieved_chunk_2}
...

Question: {question}

That "if it is not in the context, say you do not know" line is doing heavy lifting: it is what turns a confident hallucinator into a grounded assistant.

Where RAG Breaks

Retrieval, not generation, is usually the failure. When a RAG system gives a wrong answer, the model is often fine - it was simply handed the wrong chunks. Debug retrieval first: print what got fetched before blaming the model.

Semantic search misses exact terms. Embeddings are great at meaning and weak at precise identifiers - part numbers, error codes, names. The fix is hybrid search: combine vector similarity with old-fashioned keyword (BM25) search so exact matches are not lost.

The question is not shaped like the answer. A short question ("2023 refund policy?") may not embed near the long clause that answers it. Techniques like query rewriting (expand the question first) and re-ranking (fetch 30 candidates, then use a stronger model to pick the best 5) address this.

Stale or duplicated data. If your source updates, you must re-embed. Duplicated chunks crowd out diversity in the top-k. Hygiene at ingestion matters as much as the clever retrieval.

Why It Matters

RAG is how most real AI products ground themselves in private, current, or proprietary knowledge without the cost and staleness of retraining. It separates the two things a useful assistant needs - fluent language (the model) and accurate facts (your data) - and lets you improve each independently. Master retrieval quality and you have solved most of what makes AI applications actually trustworthy.