Skip to content
FORGENO ADS GAMESUnlock all $8

Prompt Engineering & AI Automation · Unit 3 of 8

Unit 3: Testing and improving prompts

The draft-test-diagnose loop, naming the kind of failure, breaking hard tasks into steps, and versioning prompts so fixes do not break old cases.

Opens the first lesson of this unit in Prompt Engineering & AI Automation. Every lesson is open from the start.

Lesson 1: The improvement loop

Goal: Run a draft-test-diagnose-change loop and keep a prompt log.

Professionals rarely write a prompt once. They run a loop: draft, test, diagnose, change one thing, test again. The loop is what turns a prompt that "seems fine" into one you can rely on.

  1. Draft the prompt with the parts from Unit 2.
  2. Test it on 5 to 10 realistic inputs, including at least two awkward ones.
  3. Diagnose: for each bad output, write one line saying what went wrong.
  4. Change one thing that targets the most common problem.
  5. Re-test on the same inputs, and keep the old version in case the change made things worse.

Worked example: the meeting-notes prompt

  1. v1: "Turn these notes into action items." Tested on 6 sets of real notes: 4 good outputs, 2 bad.
  2. Diagnosis: both bad outputs listed discussion points ("Budget was discussed") as actions.
  3. v2 adds a definition: "An action item names a person, a task and, if stated, a deadline. Ignore discussion that has no owner." Re-test on the same 6: all 6 good.
  4. v2 is saved with a note: "v2: defined action item after v1 listed discussions as actions."

One targeted change, measured on the same inputs, fixed the real problem.

Keep a simple prompt log: version, date, what changed, why, and the test result. It takes a minute and saves hours when someone asks "why does it do that?" or when a model update changes behavior and you need to know what used to work.

Prompt log
A record of each prompt version: what changed, why, and how it scored on the test inputs.
Test input
A realistic example you run a prompt on to see whether it works.

Remember: Draft, test, diagnose, change one thing, re-test on the same inputs. Log it.

Lesson 2: Diagnosing bad outputs

Goal: Name the kind of failure and pick the fix that matches it.

When an output is wrong, resist rewriting the whole prompt. Name the kind of failure first, because each kind has a typical fix.

FailureWhat it looks likeTypical fix
Misread taskSummarizes when you wanted a replyState the task first, with a clear action verb
Missing contextGeneric advice that ignores your situationAdd the facts only you know
Format driftSometimes bullets, sometimes paragraphsGive an exact template or an example output
Ignored constraint200 words when you asked for 100Make it measurable, put it near the end, give the reason
Invented detailA date or name that is not in the sourceAdd a way out (NOT FOUND) and ask for quotes
Wrong judgmentA polite complaint labeled as a questionDefine the categories and add an example of the tricky case
Too muchExtra sections, disclaimers or chatterSay exactly what to include, and to stop after it

Worked example: naming the failure

  1. Output: a reply to a customer that promises "a full refund within 24 hours".
  2. Failure type: invented detail (the policy says 5 to 7 business days), and it is also a promise the agent is not allowed to make.
  3. Fix: include the policy inside <policy> tags, add "Only state timelines that appear in the policy", and send any reply that mentions a refund to a person before it goes out.

Naming the failure points straight at the fix, and sometimes at a check the workflow needs.

Write each diagnosis as one concrete line: "Output 4 listed a discussion point as an action item" is useful; "Output 4 is bad" is not. Concrete notes add up: if three of your five notes say "format drift", you know exactly what to work on next.

Format drift
Outputs that change shape from run to run, such as switching between lists and paragraphs.

Remember: Name the failure before fixing it. Each kind of failure has its own fix.

Lesson 3: Break it down and let it think

Goal: Split multi-step tasks and keep reasoning apart from the final answer.

Some tasks hide several steps inside them: read a contract, find every deadline, compare them with today's date, decide which are urgent. Asking for the final answer in one leap invites mistakes. Two techniques help.

Spell out the steps. "First list every date in the contract with the sentence it comes from. Then mark which ones are payment deadlines. Then sort them by date." Each step is easier for the model, and when something goes wrong you can see which step failed.

Ask for reasoning before the answer. For problems that need working out, such as comparing options or checking a claim against several rules, asking the model to reason first and answer second tends to improve accuracy. Many newer models do some of this reasoning on their own. When a program reads the output, ask for the reasoning and the final answer in separate, labeled parts so your code can take just the answer.

Decide whether this expense claim follows the travel policy.
First, inside <reasoning> tags, check each policy rule against the claim.
Then, inside <answer> tags, write only APPROVE, REJECT or NEEDS REVIEW.

Watch out: Reasoning text can still be wrong, and it is not a perfect window into how the model reached its answer. Use it to spot mistakes, not as proof that the answer is right. And never rely on arithmetic the model does in its head for money: let code or a spreadsheet do the sums.

Step-by-step reasoning
Asking the model to work through a problem before giving its final answer. Often called chain of thought.

Remember: Spell out the steps. Ask for reasoning first and the answer in its own tags. Let code do the math.

Lesson 4: Versions, regressions and a prompt library

Goal: Re-run saved tests on every change and keep prompts in a shared library.

Every time you change a prompt, you risk breaking something that used to work. A fix for long emails might make short emails worse. Breaking old behavior while fixing new behavior is called a regression, and the defense is simple: keep your test inputs, and re-run all of them on every change.

Worked example: catching a regression

  1. v3 of an invoice extractor handles foreign currencies, which v2 got wrong.
  2. Re-running all 12 saved test invoices shows v3 now drops the "PO number" field on 3 invoices that v2 handled.
  3. Cause: the new currency instructions replaced part of the field list in the template.
  4. v4 restores the field list. 12 of 12 pass, including the currency cases.

Without the saved tests, v3 would have shipped with a silent bug.

Teams that use prompts at work keep them in a prompt library: one shared place (a document, a spreadsheet or a code repository) with the current version of each prompt, its owner, its settings, its test inputs and its change log. That makes prompts reusable, reviewable and safe to hand over when someone changes jobs.

Library fieldExample
NameInvoice field extractor
OwnerFinance operations
Current versionv4, 3 November 2026
Model and settingsTemperature 0, JSON output
Test set12 invoices, including 3 in foreign currencies and 2 handwritten
Change logv4: restored PO number after the v3 regression
Regression
A change that breaks something that used to work. (Statistics also uses the word for a modeling method; here it means a step backward.)
Prompt library
A shared, versioned collection of prompts with owners, tests and change logs.

Remember: Every change re-runs every saved test. Keep prompts in a library with an owner and a log.

Practise it in Prompt Engineering & AI Automation

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit.

Every lesson is free: no ads, no lives and no paywall.