Skip to content
FORGENO ADS GAMESUnlock all $8

Building AI-Powered Apps · Unit 6 of 8

Unit 6: Retrieval (RAG) and embeddings

Why apps retrieve documents, embeddings and similarity explained simply, building a retrieval pipeline with chunks and citations, and how RAG fails.

Opens the first lesson of this unit in Building AI-Powered Apps. Every lesson is open from the start.

Lesson 1: Why retrieval

Goal: Explain retrieval-augmented generation and when it beats other approaches.

A model knows only what was in its training data, up to its knowledge cutoff. It does not know your company's policies, your product manual or last week's price changes. You could paste every document into every prompt, but that is slow, expensive and limited by the context window. Retrieval-augmented generation (RAG) does something smarter: for each question, find the few passages most likely to contain the answer, put only those in the prompt, and tell the model to answer from them and cite them.

question --> search your documents --> top 3 passages --> prompt:
                                                         "Answer only from <sources>.
                                                          Cite the source. If the answer
                                                          is not there, say so."
         <-- answer with citations <--------------------------
ApproachGood forWeak at
The model aloneGeneral knowledge, writing, reasoningYour private, recent or niche facts
Paste everythingA few short documentsLarge collections; cost; buried details
RAGLarge, changing document collectionsAnswers spread across many places
Fine-tuningStyle, format and specialized behaviorFacts that change often, since retraining is slow

RAG also makes answers checkable. Because each answer comes with the passages it used, a user or reviewer can click through and confirm it, and you can measure whether the right passages were found.

Retrieval-augmented generation (RAG)
Finding relevant passages for each question and giving them to the model to answer from.
Knowledge cutoff
The date after which a model has no training data.
Grounded answer
An answer based on, and citing, provided sources.

Remember: Retrieve the few passages that matter, answer only from them, and cite them.

Lesson 2: Embeddings, explained simply

Goal: Explain embeddings and compute a dot product and cosine similarity idea.

To find relevant passages, a computer needs to measure how similar two pieces of text are. Keyword search counts shared words, which fails when people use different words for the same idea: "money back" and "refund" share no words at all. Embeddings fix that.

An embedding model turns a piece of text into a long list of numbers, a vector, positioned so that texts with similar meanings get similar vectors. "How do I get my money back?" and "What is your refund policy?" end up close together even though they share no words. Searching by embedding is called semantic search.

Worked example: similarity with tiny vectors

Real embeddings have hundreds or thousands of numbers. Imagine just three, measuring "about money", "about delivery" and "about accounts".

  1. Question "Can I get my money back?" becomes (0.9, 0.1, 0.0).
  2. Passage A, "Refunds are paid within 5 days", becomes (0.8, 0.2, 0.0). Passage B, "We deliver Monday to Saturday", becomes (0.1, 0.9, 0.1).
  3. The dot products are 0.9 × 0.8 + 0.1 × 0.2 + 0 = 0.74 for A, and 0.9 × 0.1 + 0.1 × 0.9 + 0 = 0.18 for B.
  4. Dividing by the vectors' lengths gives the cosine similarity: about 0.99 for A and 0.22 for B. Passage A wins clearly, though it never says "money back".

Embeddings let search match meaning, not just spelling.

Hands-on lab in the room: Retrieval demo. Search a small, invented company handbook two ways, by shared keywords and by a tiny built-in meaning model, and see where each succeeds and where each fails.

Embedding
A list of numbers representing a text's meaning, so similar texts get similar lists.
Vector
An ordered list of numbers.
Cosine similarity
A measure of how closely two vectors point the same way: 1 is identical direction, 0 unrelated.
Semantic search
Searching by meaning using embeddings rather than by shared words.

Remember: Embeddings turn text into vectors; similar meanings point the same way. Cosine similarity measures it.

Lesson 3: Building the pipeline

Goal: Design chunking, indexing, retrieval and a citing answer prompt.

A RAG system works in two phases.

  • Indexing, ahead of time: split documents into chunks (often a few hundred words each, keeping headings with their text, with some overlap so facts are not cut in half), compute an embedding for each chunk, and store them in a vector database or index, along with each chunk's source and who may see it.
  • Answering, for each question: embed the question, find the few most similar chunks (the top-k), optionally re-rank them with a more careful model, put them in the prompt with their source labels, and tell the model to answer only from them and cite them.
Answer the question using only the sources below. Cite the source ID for
every fact, like [S2]. If the sources do not contain the answer, say
"I could not find that in our documents" and suggest contacting HR.

<sources>
[S1] Leave policy, section 3: Staff may carry over up to 5 unused days...
[S2] Leave policy, section 4: Carried-over days must be used by 31 March...
</sources>
<question>{the staff member's question}</question>

Worked example: choosing a chunk size

  1. Whole 20-page policies as chunks: the right document is found, but the answer is buried among irrelevant text, and every request is expensive.
  2. Single sentences as chunks: precise, but "they must be used by 31 March" has lost what "they" means.
  3. Chunks of about 200 to 400 words that keep their heading, with a little overlap: each chunk makes sense on its own.

A good chunk is one a person could understand without the rest of the document.

Chunk
A passage of a document stored and retrieved as one unit.
Vector database
A store that finds the vectors most similar to a query vector quickly.
Top-k
The k most similar results returned by a search.
Re-ranking
A second, more careful pass that reorders the retrieved results.
Citation
A reference showing which source a fact came from.

Remember: Index chunks that stand alone; retrieve the top few; answer only from them, with citations and a "not found" path.

Lesson 4: When RAG goes wrong

Goal: Diagnose retrieval failures and measure retrieval separately with recall@k.

FailureExampleFix
Wrong chunk retrievedA question about returns brings back the delivery chunkBetter chunks, hybrid keyword-plus-embedding search, re-ranking
Answer not in the documentsThe model answers anyway from general knowledgeAn "only from sources" rule, a "not found" path, and checks on citations
Stale documentsThe index still holds last year's price listRe-index whenever documents change; show dates on sources
Conflicting documentsTwo versions of a policy disagreeRemove old versions; prefer the newest; flag conflicts
Permission leaksA junior employee's question retrieves the salary spreadsheetFilter chunks by the user's permissions before they reach the prompt
Injected textA web page in the index says "ignore your instructions"Treat retrieved text as data; it can never authorize actions

Measure retrieval separately from answers. Build a set of test questions, each paired with the chunk that contains its answer. Recall@k asks: was the right chunk among the top k results? If retrieval fails, no prompt can fix the answer. If retrieval works but answers are wrong, work on the prompt.

Worked example: diagnosing a bad answer

  1. Staff complain the assistant says carried-over leave expires in June. The policy says 31 March.
  2. Check retrieval first: for that question, the top 3 chunks include an old 2023 version of the policy and not the current one.
  3. Cause: the old policy was never removed from the index. Fix: re-index from the current documents only, and add the question to the retrieval test set.
  4. Recall@3 on the test set rises from 78% to 94%, and the answer is now correct with a citation to the current policy.

Check what was retrieved before blaming the model.

Recall@k
The share of test questions whose correct chunk appears in the top k retrieved results.
Hybrid search
Combining keyword search and embedding search to get the strengths of both.
Stale index
A retrieval index that still holds outdated documents.

Remember: When a RAG answer is wrong, look at what was retrieved first. Measure recall@k; filter by permissions; keep the index fresh.

Practise it in Building AI-Powered Apps

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Retrieval demo.

Every lesson is free: no ads, no lives and no paywall.