Prompt Engineering & AI Automation · Unit 3 of 8
Unit 3: Testing and improving prompts
The draft-test-diagnose loop, naming the kind of failure, breaking hard tasks into steps, and versioning prompts so fixes do not break old cases.
Opens the first lesson of this unit in Prompt Engineering & AI Automation. Every lesson is open from the start.
Lesson 1: The improvement loop
Goal: Run a draft-test-diagnose-change loop and keep a prompt log.
Professionals rarely write a prompt once. They run a loop: draft, test, diagnose, change one thing, test again. The loop is what turns a prompt that "seems fine" into one you can rely on.
- Draft the prompt with the parts from Unit 2.
- Test it on 5 to 10 realistic inputs, including at least two awkward ones.
- Diagnose: for each bad output, write one line saying what went wrong.
- Change one thing that targets the most common problem.
- Re-test on the same inputs, and keep the old version in case the change made things worse.
Worked example: the meeting-notes prompt
- v1: "Turn these notes into action items." Tested on 6 sets of real notes: 4 good outputs, 2 bad.
- Diagnosis: both bad outputs listed discussion points ("Budget was discussed") as actions.
- v2 adds a definition: "An action item names a person, a task and, if stated, a deadline. Ignore discussion that has no owner." Re-test on the same 6: all 6 good.
- v2 is saved with a note: "v2: defined action item after v1 listed discussions as actions."
One targeted change, measured on the same inputs, fixed the real problem.
Keep a simple prompt log: version, date, what changed, why, and the test result. It takes a minute and saves hours when someone asks "why does it do that?" or when a model update changes behavior and you need to know what used to work.
- Prompt log
- A record of each prompt version: what changed, why, and how it scored on the test inputs.
- Test input
- A realistic example you run a prompt on to see whether it works.
Remember: Draft, test, diagnose, change one thing, re-test on the same inputs. Log it.
Lesson 2: Diagnosing bad outputs
Goal: Name the kind of failure and pick the fix that matches it.
When an output is wrong, resist rewriting the whole prompt. Name the kind of failure first, because each kind has a typical fix.
| Failure | What it looks like | Typical fix |
|---|---|---|
| Misread task | Summarizes when you wanted a reply | State the task first, with a clear action verb |
| Missing context | Generic advice that ignores your situation | Add the facts only you know |
| Format drift | Sometimes bullets, sometimes paragraphs | Give an exact template or an example output |
| Ignored constraint | 200 words when you asked for 100 | Make it measurable, put it near the end, give the reason |
| Invented detail | A date or name that is not in the source | Add a way out (NOT FOUND) and ask for quotes |
| Wrong judgment | A polite complaint labeled as a question | Define the categories and add an example of the tricky case |
| Too much | Extra sections, disclaimers or chatter | Say exactly what to include, and to stop after it |
Worked example: naming the failure
- Output: a reply to a customer that promises "a full refund within 24 hours".
- Failure type: invented detail (the policy says 5 to 7 business days), and it is also a promise the agent is not allowed to make.
- Fix: include the policy inside <policy> tags, add "Only state timelines that appear in the policy", and send any reply that mentions a refund to a person before it goes out.
Naming the failure points straight at the fix, and sometimes at a check the workflow needs.
Write each diagnosis as one concrete line: "Output 4 listed a discussion point as an action item" is useful; "Output 4 is bad" is not. Concrete notes add up: if three of your five notes say "format drift", you know exactly what to work on next.
- Format drift
- Outputs that change shape from run to run, such as switching between lists and paragraphs.
Remember: Name the failure before fixing it. Each kind of failure has its own fix.
Lesson 3: Break it down and let it think
Goal: Split multi-step tasks and keep reasoning apart from the final answer.
Some tasks hide several steps inside them: read a contract, find every deadline, compare them with today's date, decide which are urgent. Asking for the final answer in one leap invites mistakes. Two techniques help.
Spell out the steps. "First list every date in the contract with the sentence it comes from. Then mark which ones are payment deadlines. Then sort them by date." Each step is easier for the model, and when something goes wrong you can see which step failed.
Ask for reasoning before the answer. For problems that need working out, such as comparing options or checking a claim against several rules, asking the model to reason first and answer second tends to improve accuracy. Many newer models do some of this reasoning on their own. When a program reads the output, ask for the reasoning and the final answer in separate, labeled parts so your code can take just the answer.
Decide whether this expense claim follows the travel policy.
First, inside <reasoning> tags, check each policy rule against the claim.
Then, inside <answer> tags, write only APPROVE, REJECT or NEEDS REVIEW.
Watch out: Reasoning text can still be wrong, and it is not a perfect window into how the model reached its answer. Use it to spot mistakes, not as proof that the answer is right. And never rely on arithmetic the model does in its head for money: let code or a spreadsheet do the sums.
- Step-by-step reasoning
- Asking the model to work through a problem before giving its final answer. Often called chain of thought.
Remember: Spell out the steps. Ask for reasoning first and the answer in its own tags. Let code do the math.
Lesson 4: Versions, regressions and a prompt library
Goal: Re-run saved tests on every change and keep prompts in a shared library.
Every time you change a prompt, you risk breaking something that used to work. A fix for long emails might make short emails worse. Breaking old behavior while fixing new behavior is called a regression, and the defense is simple: keep your test inputs, and re-run all of them on every change.
Worked example: catching a regression
- v3 of an invoice extractor handles foreign currencies, which v2 got wrong.
- Re-running all 12 saved test invoices shows v3 now drops the "PO number" field on 3 invoices that v2 handled.
- Cause: the new currency instructions replaced part of the field list in the template.
- v4 restores the field list. 12 of 12 pass, including the currency cases.
Without the saved tests, v3 would have shipped with a silent bug.
Teams that use prompts at work keep them in a prompt library: one shared place (a document, a spreadsheet or a code repository) with the current version of each prompt, its owner, its settings, its test inputs and its change log. That makes prompts reusable, reviewable and safe to hand over when someone changes jobs.
| Library field | Example |
|---|---|
| Name | Invoice field extractor |
| Owner | Finance operations |
| Current version | v4, 3 November 2026 |
| Model and settings | Temperature 0, JSON output |
| Test set | 12 invoices, including 3 in foreign currencies and 2 handwritten |
| Change log | v4: restored PO number after the v3 regression |
- Regression
- A change that breaks something that used to work. (Statistics also uses the word for a modeling method; here it means a step backward.)
- Prompt library
- A shared, versioned collection of prompts with owners, tests and change logs.
Remember: Every change re-runs every saved test. Keep prompts in a library with an owner and a log.
Practise it in Prompt Engineering & AI Automation
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit.
Every lesson is free: no ads, no lives and no paywall.