Responsible AI & AI Safety · Unit 1 of 8
Unit 1: What can go wrong with AI
What responsible AI means in practice, documented incidents and their lessons, the high-stakes areas where AI decisions change lives, and the environmental cost of AI.
Opens the first lesson of this unit in Responsible AI & AI Safety. Every lesson is open from the start.
Lesson 1: What responsible AI means
Goal: Describe the qualities of trustworthy AI and who is responsible for them.
An AI system is software whose behavior is learned from data rather than written as rules, so it can be wrong in ways nobody planned. Responsible AI is the practice of building, buying and using AI so that it works as intended, treats people fairly, respects privacy, can be explained and checked, stays secure, and has a person accountable for it. It is not a single tool or a slogan: it is a set of habits, documents, tests and decisions that run through a system's whole life.
The US National Institute of Standards and Technology (NIST) lists the qualities of trustworthy AI in its AI Risk Management Framework, version 1.0 of January 2023 (still the current version as of November 2026, with a revision under way). Most frameworks around the world use a similar list.
| Quality | What it means in practice |
|---|---|
| Valid and reliable | It does the job it was built for, on the people and conditions it will really meet |
| Safe | It does not put lives, health, property or the environment in danger |
| Secure and resilient | It resists attacks and recovers from failures |
| Accountable and transparent | People can find out what it is and who answers for it |
| Explainable and interpretable | People can understand how it works and why it gave a result |
| Privacy-enhanced | It collects and reveals only what it should |
| Fair, with harmful bias managed | It does not treat groups of people unjustly |
Responsibility is shared along a chain. The developer (or provider) builds the model or system. The deployer puts it to use in a real setting, such as a bank using a credit model. Users operate it day to day, and affected people live with its outputs, often without knowing AI was involved. Laws increasingly give each link its own duties: a deployer cannot shrug and say "the vendor built it".
Worked example: one system, four roles
- A software company develops a résumé-screening model and sells it.
- A hospital chain deploys it to rank applicants for nursing jobs.
- Recruiters use the ranked lists every morning.
- Applicants are affected: some are never seen by a person at all.
- If the model quietly ranks older applicants lower, the developer answers for how it was built and tested, and the hospital answers for choosing it, checking it on its own applicants and keeping a human review step.
Every role has something to check. Responsible AI work is mostly making sure each check really happens.
- Responsible AI
- Building, buying and using AI so it works as intended, is fair, private, secure, explainable and accountable.
- Trustworthy AI
- AI that has the qualities people need to rely on it, such as validity, safety, security, transparency, privacy and fairness.
- Developer (provider)
- The organization that builds an AI model or system and makes it available.
- Deployer
- The organization that uses an AI system in a real setting, under its own authority.
- Affected person
- Someone whose life is changed by an AI system's output, whether or not they use it.
Remember: Trustworthy AI is valid, safe, secure, transparent, explainable, private and fair, and someone is accountable for it.
Lesson 2: Documented AI incidents, as reported
Goal: Read real AI incidents carefully and draw the right lesson from each.
An AI incident is an event where an AI system causes or nearly causes harm. Incidents are the evidence base of this field: they show how failures actually happen, which is often not how people expected. Public collections such as the AI Incident Database and the OECD AI Incidents Monitor gather news reports so that patterns can be studied.
Read incident reports like a careful professional. Note who reported it (a newspaper, a court, a regulator, a peer-reviewed study), what was claimed, and what the organization said in response. Many cases are disputed, and a report is not a finding of guilt. This course states each case as it was reported, with its date.
| Date | What was reported | Lesson |
|---|---|---|
| May 2016 | ProPublica reported that the COMPAS risk score wrongly labeled Black defendants who did not reoffend as higher risk almost twice as often as white defendants. The vendor disputed the analysis, saying the scores predicted reoffending equally well for both groups. | Different fairness measures can disagree (Unit 2) |
| Oct 2018 | Reuters reported that Amazon dropped an experimental recruiting tool that had learned to penalize résumés containing the word "women's". Amazon said recruiters never used it to evaluate candidates. | Historical data carries historical bias |
| Oct 2019 | A study in Science found a widely used US health algorithm used past health-care cost as a stand-in for need, so it under-identified Black patients for extra care. | A proxy label can build in unfairness |
| Jan 2020 | Robert Williams was wrongfully arrested in Detroit after a facial recognition match. In June 2024 a settlement barred Detroit police from arresting people based only on face recognition results. | A match is a lead, not proof |
| Jun 2023 | A US federal judge sanctioned two lawyers who filed a brief citing court cases that ChatGPT had invented (Mata v. Avianca). | Fluent output is not verified output |
| Feb 2024 | A Canadian tribunal held Air Canada responsible for its website chatbot's wrong advice about bereavement fares (Moffatt v. Air Canada). | Organizations answer for their AI's statements |
| Feb 2024 | Hong Kong police reported that a finance employee transferred about US$25 million after a video call in which the "executives" were deepfakes. The firm was later named as Arup. | Seeing and hearing is no longer proof of identity |
Worked example: reading one report carefully
- Source: Reuters, October 2018, citing people familiar with the project.
- Claim: an experimental model trained on ten years of résumés, mostly from men, learned to downgrade signals associated with women.
- Response: Amazon said the tool was never used by its recruiters to evaluate candidates.
- Fair summary: "As reported by Reuters in 2018, Amazon abandoned an experimental hiring model after finding it penalized résumés mentioning women's activities."
- Lesson for a governance role: test models for group differences before use, because historical decisions carry historical bias.
State the source, the claim and the response, then draw the lesson. Do not overstate what was proved.
- AI incident
- An event in which an AI system causes harm or comes close to causing it.
- Near miss
- A failure that could have caused harm but was caught or did not, this time.
- AI Incident Database
- A public collection of reported AI harms, used to study patterns of failure.
- As reported
- Describing a claim with its source, without presenting it as a proven fact.
Remember: Source, claim, response, lesson: report incidents as reported, and never claim more than the evidence shows.
Lesson 3: Where the stakes are highest
Goal: Explain why hiring, lending, health, education and policing need extra care, and what goes wrong in each.
Some AI uses are low stakes: a song recommendation that misses costs a few seconds. Others decide whether someone gets a job, a loan, medical care, a fair grade or their freedom. These high-stakes (or consequential) decisions are where most AI laws focus, because errors land on people who often cannot see the system, question it or walk away.
| Area | Typical AI use | What has gone wrong | Extra care needed |
|---|---|---|---|
| Hiring | Résumé screening, ranking, video interview scoring | Models learning past hiring bias; tools screening out older applicants or disabled people | Bias audits by group, notice to candidates, human review, accommodations |
| Lending and insurance | Credit scoring, pricing, fraud flags | Proxies such as postcode standing in for race; denials nobody can explain | Specific reasons for every denial, fair lending testing, monitoring |
| Health | Risk scores, triage, imaging, chatbots | Proxy labels (cost for need); models that work worse in a new hospital | Clinical validation on local patients, clinician oversight, incident reporting |
| Education | Proctoring, essay scoring, AI-writing detectors, admissions | Grading algorithms downgrading whole schools; detectors flagging non-native English writers | No decisions on detector scores alone, appeal routes, protection of children's data |
| Policing and justice | Face recognition, risk scores, predictive policing | Wrongful arrests after false face matches; scores that mislabel groups at different rates | A match is only a lead, independent evidence, strict limits and public oversight |
In August 2020, England's exam regulator used an algorithm to estimate A-level grades when exams were cancelled. It downgraded about 39% of teachers' predicted grades, and students from historically lower-scoring schools were hit hardest; the government reversed it within days. In 2023, a peer-reviewed study (Liang and others, in Patterns) found several AI-writing detectors misclassified more than half of essays by non-native English speakers as AI-written. Both show the same pattern: a number about a group was applied to individuals who had no way to argue back.
Worked example: spotting a high-stakes use
- A gym chain wants AI to suggest class times from booking data. Wrong suggestions cost little and nobody is judged: low stakes.
- The same chain wants AI to decide which members are "likely to cancel" so staff can offer them discounts. Some people get offers others do not, but no one loses access: moderate.
- It then wants AI to flag "high-risk" members whose memberships should be refused. Now a person is denied a service by a model: high stakes, needing testing by group, a human decision-maker and a way to appeal.
The same technology moves from harmless to high stakes as soon as it decides something important about a person.
- High-stakes decision
- A decision with serious effects on a person's life, such as a job, loan, medical care, education or liberty. Also called a consequential decision.
- Proxy
- A feature or label that stands in for something else, such as cost standing in for health need, or postcode standing in for race.
- Appeal (contestability)
- A real way for an affected person to challenge a decision and have a person reconsider it.
Remember: The more an AI output decides about a person, the more testing, explanation and human review it needs.
Lesson 4: The environmental cost of AI
Goal: Estimate the energy of an AI service and weigh it honestly, with its uncertainty.
AI runs in data centres: buildings full of computers that need electricity to run and often water to cool. Two phases use energy. Training a large model is a huge one-off job; inference, answering each request, is small per request but happens billions of times. For popular services, total inference energy can exceed training.
Reported figures, with their sources: the International Energy Agency's *Energy and AI* report (April 2025) estimated that data centres used about 415 terawatt-hours (TWh) of electricity in 2024, about 1.5% of the world's electricity, and projected about 945 TWh by 2030, roughly what Japan uses today. Not all data-centre work is AI, but AI is the main driver of the growth.
Per-request figures vary widely by model, hardware and method. In August 2025 Google reported that a median text prompt in its Gemini apps used about 0.24 watt-hours, emitted about 0.03 grams of CO2-equivalent and consumed about 0.26 millilitres of water, while noting the figures were its own, not independently verified, and not true of every prompt. Image and video generation, and long reasoning answers, can use far more.
Worked example: estimating an office assistant's yearly energy
- Assume (an invented figure for practice) 0.3 Wh per request.
- 400 staff each make 25 requests a day: 400 × 25 = 10,000 requests a day.
- Per day: 10,000 × 0.3 Wh = 3,000 Wh = 3 kWh.
- Per year of 250 working days: 3 × 250 = 750 kWh, roughly what a small fridge-freezer uses in two years.
- Now compare: the same work might replace hours of searching, or it might add unneeded calls to a large model when a small one, or no AI, would do.
Energy = requests × energy per request. The honest answer includes the assumption, the range and what the AI replaces.
Responsible choices include: use the smallest model that does the job, avoid generating what nobody needs, cache repeated answers, prefer providers that publish energy, carbon and water data, and put environmental impact in the risk assessment alongside other harms. Be wary of both extremes: headlines that one question "boils a kettle", and claims that the cost is negligible at any scale.
- Data centre
- A building of networked computers that runs online services, including AI.
- Training (energy)
- The large one-off computation that produces a model's parameters.
- Inference (energy)
- The computation used each time a trained model answers a request.
- Watt-hour (Wh)
- A unit of energy: one watt used for one hour. 1,000 Wh = 1 kilowatt-hour (kWh).
- Carbon intensity
- How much CO2 is emitted per unit of electricity, which depends on how the power was generated.
Remember: Total footprint = number of requests × energy per request, plus training. State assumptions and ranges.
Practise it in Responsible AI & AI Safety
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit.
Every lesson is free: no ads, no lives and no paywall.