RAG is not magic: building retrieval-augmented generation from zero

Hello, and welcome. This article will talk about retrieval-augmented generation, or RAG, and why an LLM gives confident wrong answers about your own documents.

By the end of this article, you will understand the three steps that make up RAG, be able to build a working version in about 100 lines of Python, and know where the simple version breaks and how to fix it.

Your LLM doesn’t know your documents. It was trained months ago on the public internet. It has never seen your company handbook, your thesis PDFs, or last week’s meeting notes. RAG is a way of showing them to it at the moment you ask a question.

I spend my research time on exactly this. RAG is three small, understandable steps glued together. Let’s build them from zero.

Frozen knowledge, a small window, and an expensive alternative

An LLM has two hard limits. Its knowledge is frozen at training time. And it can only read so much text at once (the context window). You could re-train or fine-tune it on your documents. But that’s expensive and slow, and you have to redo it every time a document changes. RAG avoids all of that. Don’t put your knowledge into the model. Retrieve the relevant pieces and hand them to the model along with the question.

Turn meaning into coordinates

The first trick is turning text into numbers that capture meaning. An embedding model maps a piece of text to a vector. A vector is a point in a space with a few hundred dimensions. Text with similar meaning lands nearby. “Bank account” and “savings” become neighbours. “Savings” and “river bank” do not.

Once everything is a point, “find relevant text” becomes “find nearby points”. That is a geometry problem, measured with cosine similarity.

How you cut the document matters more than people think

You can’t embed a whole 80-page PDF as one vector. You’d average away all the detail. So you split documents into chunks, embed each one, and store the vectors. How you chunk decides how good your retrieval can ever be. Split on a blind fixed character count and you’ll cut sentences in half, even key terms. You end up embedding two halves of an idea that mean nothing on their own.

The whole loop, end to end

At query time, you embed the question with the same model. You find the top-k nearest chunks. You paste them into the prompt as context. Then you ask the model to answer using only that context, with citations.

---

No framework, so you see every moving part

Here is a minimal, working RAG over your own PDFs. It uses no framework on purpose, so nothing is hidden:

import numpy as np
from openai import OpenAI            # any embedding + chat API works

client = OpenAI()

def embed(texts: list[str]) -> np.ndarray:
    out = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in out.data])

def chunk(text: str, size=800, overlap=150) -> list[str]:
    # split on paragraphs, then pack into ~size-char chunks with overlap
    paras, chunks, buf = text.split("\n\n"), [], ""
    for p in paras:
        if len(buf) + len(p) > size:
            chunks.append(buf.strip()); buf = buf[-overlap:]
        buf += "\n\n" + p
    if buf.strip(): chunks.append(buf.strip())
    return chunks

# --- index (once) ---
chunks = chunk(open("thesis.txt").read())
vectors = embed(chunks)              # store these; a real app uses a vector DB

def retrieve(question: str, k=4) -> list[str]:
    q = embed([question])[0]
    sims = vectors @ q / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q))
    top = np.argsort(sims)[::-1][:k]
    return [chunks[i] for i in top]

def answer(question: str) -> str:
    context = "\n\n---\n\n".join(retrieve(question))
    prompt = (f"Answer using ONLY the context. Cite chunk numbers. "
              f"If the context doesn't cover it, say so.\n\nContext:\n{context}\n\nQ: {question}")
    resp = client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}])
    return resp.choices[0].message.content

print(answer("What evaluation metric did the thesis use?"))

The whole pipeline is chunk, embed, retrieve by cosine similarity, fill the prompt, generate. Everything else in the RAG ecosystem is an improvement on top of these four functions.

And the fixes that earn their keep

The version above works in a demo and disappoints in production. It fails in predictable ways.

• Bad chunks: poor splitting retrieves half-thoughts. Fix: better chunking (above), or Anthropic’s contextual retrieval, which adds a short context blurb to the front of each chunk before embedding.
• Wrong retrieval: the nearest vectors aren’t always the most useful. Fix: hybrid search (combine semantic similarity with old-fashioned keyword/BM25 search) and reranking (a second model re-scores the top candidates).
• Hallucinated citations: the model cites a chunk that doesn’t support the claim. Fix: insist on quote-then-answer, and check that the cited text really exists.
• Bad question: a vague query embeds to a vague place. Fix: query rewriting before retrieval.

Pick the tool for the knowledge problem

These three solve different problems. RAG adds changing, citable facts. Fine-tuning teaches behaviour and style. A long context window is great for one big document you have right now. But it pays the token cost on every call and forgets the document afterwards.

Which one does your problem need?

Recap

The goal of this article was to show that RAG is three understandable steps, not magic, and to give you a working version you can build and debug yourself.

• An LLM’s knowledge is frozen and its context window is small. RAG fixes this by retrieving the relevant pieces of your documents and handing them to the model with the question.
• Embeddings turn text into points in space, so “find relevant text” becomes “find nearby points” using cosine similarity.
• How you chunk decides how good retrieval can ever be. Split on natural boundaries and add a little overlap.
• The whole pipeline is four functions: chunk, embed, retrieve, then generate. Everything else is an improvement on top.
• When RAG fails, it is usually a retrieval problem. Print the chunks before you blame the model.
• RAG, fine-tuning, and long context solve different problems. RAG is for changing, citable facts. Fine-tuning is for behaviour and style. Long context is for one document you have right now.

RAG earned its place because knowledge changes and models are expensive to retrain. Understand those four functions and you can debug any RAG system, framework or not, because you’ll know which moving part went wrong.

I hope this writing helps you. Thanks for reading, and if it cleared something up, share it with someone who is building their first RAG.