Skip to content
FORGENO ADS GAMESUnlock all $8

Building AI-Powered Apps · Unit 8 of 8

Unit 8: Evaluating, shipping and privacy

Evaluating an AI feature before launch, launching and operating it with monitoring and budgets, designing for privacy, and the checklist professionals use.

Opens the first lesson of this unit in Building AI-Powered Apps. Every lesson is open from the start.

Lesson 1: Evaluate before you ship

Goal: Test components and the whole app against agreed launch criteria.

An AI feature is ready when you can show, with numbers, that it does its job well enough and fails safely. Build an evaluation set for the app's real tasks, and test each part as well as the whole.

What you testExample check
RetrievalIs the right chunk in the top 3? (recall@3)
Tool choiceDoes "where is my order?" call lookup_order with the right argument?
FormatIs every structured output valid against the schema?
Answer qualityA rubric: correct, grounded, within policy, right tone
SafetyDo all the red-team cases fail safely?
Cost and speedAverage tokens and seconds per conversation

Agree launch criteria in advance, for example: at least 95% of answers graded correct and grounded on the test set, zero critical safety failures, and an average cost per conversation under budget. Re-run the whole set on every prompt change, model change or new data source, and block the release if anything regresses.

Worked example: a launch decision

  1. Results on 120 test conversations: 96% correct and grounded; recall@3 92%; all 25 red-team cases handled safely; average cost within budget.
  2. One problem: 3 answers about warranty periods cite the right document but state the wrong number of years.
  3. The team adds a rule ("quote warranty periods exactly as written"), re-runs all 120 cases and gets 99%.
  4. The criteria are met. They launch to staff first (next lesson).

Numbers agreed in advance turn "is it ready?" into a check, not an argument.

Launch criteria
Measurable conditions, agreed in advance, that a feature must meet before release.
Component evaluation
Testing one part, such as retrieval or tool choice, on its own.
End-to-end evaluation
Testing the whole app from question to final answer.

Remember: Test retrieval, tools, format, quality, safety and cost. Agree launch criteria first; block releases that regress.

Lesson 2: Launch and operate

Goal: Plan a staged rollout with monitoring, budget alerts and fallbacks, and forecast monthly cost.

  • Staged rollout: start with internal staff, then 5% of users, then more, watching the numbers at each stage.
  • Feature flags: switch the feature on or off, or change the model, without redeploying.
  • Monitoring: error rates, latency, tokens and cost per day, user feedback (thumbs up or down) and hand-offs to people.
  • Budget alerts: an alert at 80% of the monthly budget, and a hard cap.
  • Fallbacks: if the provider is down, show a helpful message and a way to reach a person; some apps can switch to a second provider.
  • Feedback loop: review thumbs-down answers weekly and turn them into test cases.

Hands-on lab in the room: Simulated model API and budget planner. Plan the monthly cost of a feature: users, conversations, tokens per turn, model choice, routing and caching.

Worked example: a budget forecast

2,000 users, each with about 5 conversations a month of 4 turns; each turn about 1,500 input and 250 output tokens. Invented prices: large model 3 and 15 dollars per million tokens, small model 1 and 5.

  1. Turns per month: 2,000 × 5 × 4 = 40,000. Input: 40,000 × 1,500 = 60 million tokens; output: 40,000 × 250 = 10 million.
  2. All on the large model: 60 × 3 + 10 × 15 = 180 + 150 = 330 dollars a month.
  3. All on the small model: 60 × 1 + 10 × 5 = 110 dollars a month.
  4. Routing 70% of turns to the small model and 30% to the large: 0.7 × 110 + 0.3 × 330 = 77 + 99 = 176 dollars a month.

A one-page forecast like this, with an alert and a cap, prevents nasty surprises.

Staged rollout
Releasing a feature to a small group first, then widening it as the numbers hold up.
Feature flag
A setting that turns a feature on or off without a new deployment.
Budget alert
A notification when spending reaches a set share of the budget.

Remember: Roll out in stages behind a flag, watch errors, cost and feedback, alert at 80% of budget, and have a fallback.

Lesson 3: Privacy by design

Goal: Minimize, redact and retain data responsibly, and tell users how their data is used.

Every request sends data to a third party, the model provider, and your logs keep copies. Privacy has to be designed in from the start.

  • Minimize: send only what the task needs. A delivery question does not need the customer's date of birth.
  • Redact: remove card numbers, ID numbers and other sensitive details before sending, and before logging.
  • Know your provider's terms: whether it keeps request data, for how long, and whether it may be used for training; choose settings and contracts that fit your obligations.
  • Tell users: say in plain words that an AI provider processes their messages, and what is kept.
  • Retention: delete logs and conversation history on a schedule, and honor deletion requests.
  • Sensitive data and children: health, financial and children's data carry extra legal duties in many places, for example under the GDPR in Europe and various US laws. Get expert advice before building in these areas.

Worked example: minimizing one request

  1. Before: the context includes the customer's full profile: name, date of birth, home address, phone, email, every order since 2019 and saved payment details.
  2. The question is "Where is my order 5512?"
  3. After: the context includes the customer's first name and order 5512's status and tracking number. Nothing else.
  4. The answer is just as good, the request is cheaper, and far less personal data leaves your systems.

Data you never send cannot leak from someone else's logs.

Data minimization
Collecting and sending only the personal data a task actually needs.
Retention period
How long data is kept before it is deleted.
Personal data (PII)
Information that identifies, or could identify, a person.

Remember: Send the minimum, redact the sensitive, know the provider's terms, tell users, delete on schedule.

Lesson 4: Putting it all together

Goal: Review an AI feature against a professional launch checklist.

You now know the moving parts of an AI-powered app. Here they are as one checklist, the kind a reviewer uses before approving a launch.

An AI feature launch checklist
AreaQuestion
ArchitectureDoes the browser only talk to your server, with keys in environment variables?
CostDo you know the cost per conversation and per month, with alerts and a cap?
ReliabilityDo you handle 429s and errors with backoff, timeouts and friendly fallbacks?
PromptsIs the system prompt versioned, tested and free of secrets?
OutputsIs every structured output validated, and every reply rendered as text?
ToolsDo tools run with the user's permissions, with confirmation for consequential actions?
RetrievalIs retrieval measured, filtered by permissions and kept fresh?
SafetyAre there input and output guardrails, red-team tests and a kill switch?
PrivacyIs data minimized and redacted, retention set, and users told?
EvaluationAre there launch criteria and a regression run on every change?

In interviews for roles that build with AI, often called AI engineer, applied AI developer or LLM application developer, expect questions on exactly these trade-offs: how you would control cost, how you would measure quality, how you would stop prompt injection from causing harm. A small, well-documented project that shows them, like this course's final project, says more than a list of tools.

This course teaches the foundations and the professional habits. Building production apps also needs general programming, web development and security skills. Code Academy and Cyber Academy on this site are good companions.

AI engineer
A developer who builds products on top of AI models, covering prompts, retrieval, tools, evaluation and safety.

Remember: Architecture, cost, reliability, prompts, outputs, tools, retrieval, safety, privacy, evaluation: check all ten.

Practise it in Building AI-Powered Apps

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Simulated model API and budget planner.

Every lesson is free: no ads, no lives and no paywall.