RAG is not magic: building retrieval-augmented generation from zero

Your LLM doesn’t know your documents. It was trained months ago on the public internet, and it has never seen your company handbook, your thesis PDFs, or last week’s meeting notes. Retrieval-augmented generation is a way of showing them to it at the moment you ask a question, nothing more and nothing less. Once you see it that plainly, the architecture explains itself.

I spend my research time on exactly this, and most confusion about RAG comes from treating it as one mysterious thing. It isn’t. It’s three small, understandable steps glued together. Let’s build them from zero.

Frozen knowledge, a small window, and an expensive alternative

An LLM has two hard limits: its knowledge is frozen at training time, and it can only read so much text at once (the context window). You could re-train or fine-tune it on your documents, but that’s expensive, slow, and has to be redone every time a document changes. RAG sidesteps all of that: don’t put your knowledge into the model. Retrieve the relevant pieces and hand them to the model alongside the question.

Turn meaning into coordinates

The first trick is turning text into numbers that capture meaning. An embedding model maps a piece of text to a vector, a point in a few-hundred-dimensional space, such that text with similar meaning lands nearby. “Bank account” and “savings” become neighbours; “savings” and “river bank” do not.

Once everything is a point, “find relevant text” becomes “find nearby points”, a geometry problem measured with cosine similarity.

How you cut the document matters more than people think

You can’t embed a whole 80-page PDF as one vector; you’d average away all the detail. So you split documents into chunks, embed each, and store the vectors. The catch nobody warns you about: how you chunk decides how good your retrieval can ever be. Split on a blind fixed character count and you’ll slice sentences, even key terms, in half, embedding two halves of an idea that mean nothing apart.

The whole loop, end to end

At query time: embed the question with the same model, find the top-k nearest chunks, paste them into the prompt as context, and ask the model to answer using only that context, with citations.

---

No framework, so you see every moving part

Here is a minimal, working RAG over your own PDFs, deliberately framework-free so nothing is hidden:

import numpy as np
from openai import OpenAI            # any embedding + chat API works

client = OpenAI()

def embed(texts: list[str]) -> np.ndarray:
    out = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in out.data])

def chunk(text: str, size=800, overlap=150) -> list[str]:
    # split on paragraphs, then pack into ~size-char chunks with overlap
    paras, chunks, buf = text.split("\n\n"), [], ""
    for p in paras:
        if len(buf) + len(p) > size:
            chunks.append(buf.strip()); buf = buf[-overlap:]
        buf += "\n\n" + p
    if buf.strip(): chunks.append(buf.strip())
    return chunks

# --- index (once) ---
chunks = chunk(open("thesis.txt").read())
vectors = embed(chunks)              # store these; a real app uses a vector DB

def retrieve(question: str, k=4) -> list[str]:
    q = embed([question])[0]
    sims = vectors @ q / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q))
    top = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in top]

def answer(question: str) -> str:
    context = "\n\n---\n\n".join(retrieve(question))
    prompt = (f"Answer using ONLY the context. Cite chunk numbers. "
              f"If the context doesn't cover it, say so.\n\nContext:\n{context}\n\nQ: {question}")
    resp = client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}])
    return resp.choices[0].message.content

print(answer("What evaluation metric did the thesis use?"))

That’s the whole idea: chunk, embed, retrieve by cosine similarity, stuff the prompt, generate. Everything else in the RAG ecosystem is an optimisation on top of these four functions.

And the fixes that earn their keep

The version above works in a demo and disappoints in production, in predictable ways.

• Bad chunks: junk splitting retrieves half-thoughts. Fix: better chunking (above), or Anthropic’s contextual retrieval, which prepends a short context blurb to each chunk before embedding.
• Wrong retrieval: the nearest vectors aren’t always the most useful. Fix: hybrid search (combine semantic similarity with old-fashioned keyword/BM25 search) and reranking (a second model re-scores the top candidates).
• Hallucinated citations: the model cites a chunk that doesn’t support the claim. Fix: insist on quote-then-answer, and verify cited spans exist.
• Bad question: a vague query embeds to a vague place. Fix: query rewriting before retrieval.

Pick the tool for the knowledge problem

These three are often framed as rivals; they solve different problems. RAG injects changing, citable facts. Fine-tuning teaches behaviour and style. A long context window is great for one big document you have right now but pays the token cost every call and forgets it afterwards.

Which one does your problem need?

RAG earned its place because knowledge changes and models are expensive to retrain. But it is not magic: it is chunking, embeddings, and a similarity search, wired to a prompt. Understand those four functions and you can debug any RAG system, framework or not, because you’ll know which moving part went wrong.