Building AI-Powered Apps · Unit 6 of 8
Unit 6: Retrieval (RAG) and embeddings
Why apps retrieve documents, embeddings and similarity explained simply, building a retrieval pipeline with chunks and citations, and how RAG fails.
Opens the first lesson of this unit in Building AI-Powered Apps. Every lesson is open from the start.
Lesson 1: Why retrieval
Goal: Explain retrieval-augmented generation and when it beats other approaches.
A model knows only what was in its training data, up to its knowledge cutoff. It does not know your company's policies, your product manual or last week's price changes. You could paste every document into every prompt, but that is slow, expensive and limited by the context window. Retrieval-augmented generation (RAG) does something smarter: for each question, find the few passages most likely to contain the answer, put only those in the prompt, and tell the model to answer from them and cite them.
question --> search your documents --> top 3 passages --> prompt:
"Answer only from <sources>.
Cite the source. If the answer
is not there, say so."
<-- answer with citations <--------------------------
| Approach | Good for | Weak at |
|---|---|---|
| The model alone | General knowledge, writing, reasoning | Your private, recent or niche facts |
| Paste everything | A few short documents | Large collections; cost; buried details |
| RAG | Large, changing document collections | Answers spread across many places |
| Fine-tuning | Style, format and specialized behavior | Facts that change often, since retraining is slow |
RAG also makes answers checkable. Because each answer comes with the passages it used, a user or reviewer can click through and confirm it, and you can measure whether the right passages were found.
- Retrieval-augmented generation (RAG)
- Finding relevant passages for each question and giving them to the model to answer from.
- Knowledge cutoff
- The date after which a model has no training data.
- Grounded answer
- An answer based on, and citing, provided sources.
Remember: Retrieve the few passages that matter, answer only from them, and cite them.
Lesson 2: Embeddings, explained simply
Goal: Explain embeddings and compute a dot product and cosine similarity idea.
To find relevant passages, a computer needs to measure how similar two pieces of text are. Keyword search counts shared words, which fails when people use different words for the same idea: "money back" and "refund" share no words at all. Embeddings fix that.
An embedding model turns a piece of text into a long list of numbers, a vector, positioned so that texts with similar meanings get similar vectors. "How do I get my money back?" and "What is your refund policy?" end up close together even though they share no words. Searching by embedding is called semantic search.
Worked example: similarity with tiny vectors
Real embeddings have hundreds or thousands of numbers. Imagine just three, measuring "about money", "about delivery" and "about accounts".
- Question "Can I get my money back?" becomes (0.9, 0.1, 0.0).
- Passage A, "Refunds are paid within 5 days", becomes (0.8, 0.2, 0.0). Passage B, "We deliver Monday to Saturday", becomes (0.1, 0.9, 0.1).
- The dot products are 0.9 × 0.8 + 0.1 × 0.2 + 0 = 0.74 for A, and 0.9 × 0.1 + 0.1 × 0.9 + 0 = 0.18 for B.
- Dividing by the vectors' lengths gives the cosine similarity: about 0.99 for A and 0.22 for B. Passage A wins clearly, though it never says "money back".
Embeddings let search match meaning, not just spelling.
Hands-on lab in the room: Retrieval demo. Search a small, invented company handbook two ways, by shared keywords and by a tiny built-in meaning model, and see where each succeeds and where each fails.
- Embedding
- A list of numbers representing a text's meaning, so similar texts get similar lists.
- Vector
- An ordered list of numbers.
- Cosine similarity
- A measure of how closely two vectors point the same way: 1 is identical direction, 0 unrelated.
- Semantic search
- Searching by meaning using embeddings rather than by shared words.
Remember: Embeddings turn text into vectors; similar meanings point the same way. Cosine similarity measures it.
Lesson 3: Building the pipeline
Goal: Design chunking, indexing, retrieval and a citing answer prompt.
A RAG system works in two phases.
- Indexing, ahead of time: split documents into chunks (often a few hundred words each, keeping headings with their text, with some overlap so facts are not cut in half), compute an embedding for each chunk, and store them in a vector database or index, along with each chunk's source and who may see it.
- Answering, for each question: embed the question, find the few most similar chunks (the top-k), optionally re-rank them with a more careful model, put them in the prompt with their source labels, and tell the model to answer only from them and cite them.
Answer the question using only the sources below. Cite the source ID for
every fact, like [S2]. If the sources do not contain the answer, say
"I could not find that in our documents" and suggest contacting HR.
<sources>
[S1] Leave policy, section 3: Staff may carry over up to 5 unused days...
[S2] Leave policy, section 4: Carried-over days must be used by 31 March...
</sources>
<question>{the staff member's question}</question>
Worked example: choosing a chunk size
- Whole 20-page policies as chunks: the right document is found, but the answer is buried among irrelevant text, and every request is expensive.
- Single sentences as chunks: precise, but "they must be used by 31 March" has lost what "they" means.
- Chunks of about 200 to 400 words that keep their heading, with a little overlap: each chunk makes sense on its own.
A good chunk is one a person could understand without the rest of the document.
- Chunk
- A passage of a document stored and retrieved as one unit.
- Vector database
- A store that finds the vectors most similar to a query vector quickly.
- Top-k
- The k most similar results returned by a search.
- Re-ranking
- A second, more careful pass that reorders the retrieved results.
- Citation
- A reference showing which source a fact came from.
Remember: Index chunks that stand alone; retrieve the top few; answer only from them, with citations and a "not found" path.
Lesson 4: When RAG goes wrong
Goal: Diagnose retrieval failures and measure retrieval separately with recall@k.
| Failure | Example | Fix |
|---|---|---|
| Wrong chunk retrieved | A question about returns brings back the delivery chunk | Better chunks, hybrid keyword-plus-embedding search, re-ranking |
| Answer not in the documents | The model answers anyway from general knowledge | An "only from sources" rule, a "not found" path, and checks on citations |
| Stale documents | The index still holds last year's price list | Re-index whenever documents change; show dates on sources |
| Conflicting documents | Two versions of a policy disagree | Remove old versions; prefer the newest; flag conflicts |
| Permission leaks | A junior employee's question retrieves the salary spreadsheet | Filter chunks by the user's permissions before they reach the prompt |
| Injected text | A web page in the index says "ignore your instructions" | Treat retrieved text as data; it can never authorize actions |
Measure retrieval separately from answers. Build a set of test questions, each paired with the chunk that contains its answer. Recall@k asks: was the right chunk among the top k results? If retrieval fails, no prompt can fix the answer. If retrieval works but answers are wrong, work on the prompt.
Worked example: diagnosing a bad answer
- Staff complain the assistant says carried-over leave expires in June. The policy says 31 March.
- Check retrieval first: for that question, the top 3 chunks include an old 2023 version of the policy and not the current one.
- Cause: the old policy was never removed from the index. Fix: re-index from the current documents only, and add the question to the retrieval test set.
- Recall@3 on the test set rises from 78% to 94%, and the answer is now correct with a citation to the current policy.
Check what was retrieved before blaming the model.
- Recall@k
- The share of test questions whose correct chunk appears in the top k retrieved results.
- Hybrid search
- Combining keyword search and embedding search to get the strengths of both.
- Stale index
- A retrieval index that still holds outdated documents.
Remember: When a RAG answer is wrong, look at what was retrieved first. Measure recall@k; filter by permissions; keep the index fresh.
Practise it in Building AI-Powered Apps
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Retrieval demo.
Every lesson is free: no ads, no lives and no paywall.