Skip to content
FORGENO ADS GAMESUnlock all $8

Responsible AI & AI Safety · Unit 8 of 8

Unit 8: Risk, incidents, policy and careers

Scoring risks in a register, impact assessments, responding to and reporting AI incidents, writing an AI use policy, the field's open debates, and building a career in it.

Opens the first lesson of this unit in Responsible AI & AI Safety. Every lesson is open from the start.

Lesson 1: Risk and impact assessment

Goal: Score AI risks by likelihood and impact, choose mitigations and record residual risk in a register.

A risk is something that might happen and would cause harm. Most organizations rate each risk on two scales, likelihood (how probable, 1 = rare to 5 = almost certain) and impact (how bad, 1 = minor to 5 = severe), and multiply them: risk score = likelihood × impact, from 1 to 25. Bands turn the number into action. This course uses: 1–4 low (accept and monitor), 5–9 medium (mitigate where practical), 10–16 high (mitigate before launch), 20–25 critical (do not launch until reduced).

The score before any controls is the inherent risk. Mitigations reduce likelihood (a human review step makes a wrong decision less likely to stand) or impact (a fast appeal limits the damage). The score after them is the residual risk. Some residual risk always remains; the question is whether it is within the organization's risk appetite, and an accountable person with the authority to do so must accept it in writing. Every risk needs an owner and a review date.

Worked example: three risks for an invented tenant-screening tool

  1. Risk A: applicants wrongly rejected because eviction records are matched to the wrong person with a similar name. Evidence: name-matching errors are common in such data. Likelihood 4, impact 4: score 16, high.
  2. Mitigation for A: require a match on date of birth as well as name, and let applicants dispute records before any decision. Likelihood falls to 2: residual 2 × 4 = 8, medium.
  3. Risk B: rejection rates differ sharply by race because past evictions reflect past discrimination. Likelihood 3, impact 5: score 15, high. Mitigation: test impact ratios before launch and quarterly, and remove eviction filings that never led to a judgment. Likelihood 2: residual 10, still high; add human review of every rejection, likelihood 1: residual 5, medium.
  4. Risk C: the screening report is slow on Mondays. Likelihood 3, impact 1: score 3, low: accept and monitor.
  5. The register records each risk, its owner, scores before and after, mitigations, who accepted the residual risk and the next review date.

Score = likelihood × impact. Mitigate until residual risk is within appetite, and record who accepted what remains.

Risk registers focus on harm to the organization and to people. Impact assessments look harder at people and society: who could be affected, how their rights could be harmed, whether some groups bear more of the cost, and what the environmental cost is. Examples include the data protection impact assessment (Unit 3), ISO/IEC 42005 guidance on AI system impact assessments, the EU AI Act's fundamental rights impact assessment for some deployers, and Canada's government Algorithmic Impact Assessment, a questionnaire that assigns one of four impact levels and scales the required safeguards to it.

Hands-on lab in the room: AI risk register. Score five risks for an invented city benefits chatbot from the evidence, give each an owner, and choose mitigations until every residual risk is medium or below. Some mitigations sound good and do nothing.

Risk register
A living list of risks with their scores, owners, mitigations, residual scores and review dates.
Likelihood
How probable it is that a risk will happen, usually scored 1 to 5.
Impact (of a risk)
How severe the harm would be if a risk happened, usually scored 1 to 5.
Inherent risk
The level of a risk before any controls or mitigations.
Residual risk
The level of a risk that remains after mitigations.
Risk owner
The named person responsible for managing a risk and reporting on it.
Risk appetite
The amount and kind of risk an organization is willing to accept to reach its goals.
Algorithmic impact assessment
A structured assessment of how an automated system could affect people's rights and interests, and what safeguards it needs.

Remember: Likelihood × impact, before and after mitigation. Every risk gets an owner, and someone with authority accepts what remains.

Lesson 2: Incident response and reporting

Goal: Run the response to an AI incident and write a report with every required field.

An AI incident (Unit 1) needs a planned response, not improvisation. The usual life cycle: prepare (owners, a playbook, logs and a way to switch the feature off); detect (monitoring, user complaints, staff reports, red-team findings); triage (how severe, how widespread); contain (stop the harm first: roll back, disable the feature, switch to a human process); investigate (find the root cause in the data, model, prompt, integration or human process); remediate; notify and report (affected people, regulators, partners, as required); and learn (a blameless post-incident review that updates tests, the risk register and training).

An example severity scale. Organizations define their own.
SeverityExampleTypical response
SEV 1, criticalAn AI system is causing or could cause death, serious injury or serious rights violations, or is leaking sensitive data at scaleContain immediately, executive and legal involvement, regulator reporting checked within hours
SEV 2, highMany people are getting wrong decisions or harmful advice, or a security control has failedContain the same day, notify affected people, check reporting duties
SEV 3, mediumErrors affect a few people, with workarounds availableFix in the normal cycle, monitor, log for patterns
SEV 4, lowCosmetic or minor errors, no one harmedLog and fix when convenient

Some incidents must be reported to authorities, with deadlines. As of November 2026: under the EU AI Act (Article 73), providers of high-risk systems must report serious incidents within 15 days of becoming aware, within 10 days if a person died, and within 2 days for a widespread infringement or serious disruption of critical infrastructure, once the high-risk duties apply; providers of general-purpose models with systemic risk report serious incidents to the EU AI Office. Under the GDPR, a personal data breach goes to the data protection authority within 72 hours. California's SB 53 requires frontier developers to report critical safety incidents within 15 days, or 24 hours if there is an imminent risk of death or serious injury; New York's RAISE Act will require 72 hours from 2027. Sector regulators, contracts and insurers may add their own rules.

Worked example: a benefits chatbot gives wrong eligibility advice

  1. Detect: on 2026-03-09 a caseworker notices three applicants who were told by an invented city's chatbot that they "do not qualify" for a food benefit they do qualify for.
  2. Triage: logs show 212 conversations with the same wrong answer since a policy update on 2026-03-02. People may have given up applying: SEV 2.
  3. Contain: the eligibility topic is switched off in the chatbot the same day; a banner tells users to call the helpline.
  4. Investigate: root cause: the new income threshold was published as a PDF the chatbot's retrieval system could not read, so it used the old threshold.
  5. Remediate and notify: the document is fixed and a test is added for every threshold; the city contacts the 212 people, using the logs, and invites them to apply.
  6. Report and learn: the data protection officer confirms no personal data breach occurred, so no GDPR-style notice is needed; the review adds "policy change" as a trigger for re-testing.

Contain first, find the root cause, put it right for the people harmed, report where required, and change the process so it cannot happen the same way again.

Hands-on lab in the room: Incident report writer. Write an incident report for an invented case. The grader checks every required field and flags guesses presented as facts.

Serious incident (EU AI Act)
An incident or malfunction of an AI system that leads to death or serious harm to health, serious disruption of critical infrastructure, infringement of fundamental rights, or serious harm to property or the environment.
Severity level
A rating of how serious an incident is, which sets how fast and how widely the organization responds.
Containment
The first actions that stop an incident's harm from spreading, such as disabling a feature.
Root cause analysis
Investigating past the symptoms to the underlying reason an incident happened.
Post-incident review
A meeting and report after an incident to learn lessons and change processes, ideally blameless.
Kill switch
A prepared way to turn off an AI feature or system quickly.

Remember: Prepare, detect, triage, contain, investigate, remediate, report, learn. Contain before you investigate, and know your deadlines.

Lesson 3: Writing an AI use policy

Goal: Draft a clear, practical AI use policy for a small organization.

Staff are already using AI, whether or not there is a policy: chatbots in a browser tab, AI features inside office software, transcription in video calls. Use the organization has not approved or does not know about is called shadow AI. An AI use policy tells everyone which tools they may use, for what, with which data, who checks the results and where to go with questions. For a small organization it should fit on two pages and use plain words.

SectionWhat it saysExample for a small organization
Purpose and scopeWhy the policy exists and who it coversAll staff, volunteers and contractors; all AI tools, including AI features inside existing software
Approved toolsWhich tools may be used, and how to ask for a new oneOne business AI assistant with a no-training agreement; new tools go to the policy owner for a short review
Data rulesWhat may and may not go into AI toolsGreen: public information. Amber: internal documents, approved tools only. Red: personal, health, financial or client data, never in any AI tool without written approval
Allowed and banned usesWhat AI may help with, and what it must never doDrafting, summarizing and brainstorming are fine. Never: final decisions about people, deepfakes of real people, or advice presented as professional judgment without review
Human review and accountabilityWho checks output, and who answers for itThe person who uses the tool is responsible for the result and must check facts, figures and tone before anything goes out
DisclosureWhen to tell people AI was usedTell people when they are chatting with AI; label AI-generated images; follow funders' and clients' rules
Training, incidents and reviewHow people learn, report problems and keep the policy currentA short AI literacy session for everyone; report problems to the policy owner within one working day; review every six months

Good policies are specific ("never paste client names into an AI tool"), not vague ("use AI responsibly"); they explain why, so people can handle cases the policy did not foresee; they name an owner; and they make the safe path easy, for example by providing an approved tool so people are not tempted to use personal accounts. They also connect to everything else in this course: data protection, the risk register, incident response, procurement and vendor review.

Worked example: four clauses for an invented food bank with 18 staff

  1. Data: "Never enter the name, address, phone number, health details or immigration status of any client or volunteer into an AI tool. Use initials or remove details first."
  2. Uses: "AI may help draft newsletters, grant applications and rotas. It may not decide who receives food parcels or who is offered a volunteer role."
  3. Review: "Whoever uses AI checks every fact, figure and name before anything is sent or published. You are responsible for the final text."
  4. Owner and review: "The operations manager owns this policy, keeps the list of approved tools, and reviews the policy every six months or after any incident."

Specific rules, reasons people understand, an approved tool, a named owner and a review date.

Hands-on lab in the room: AI use-policy drafter. Draft an AI use policy for an invented community health clinic. Choose an option for each section; weak options are explained, and the finished policy can be copied.

AI use policy
An organization's rules for which AI tools staff may use, for what, with which data, and with what review.
Shadow AI
Use of AI tools that an organization has not approved or does not know about.
Acceptable use
The uses of a tool or system that an organization permits.
Data classification
Sorting information into levels, such as public, internal and confidential, with rules for each.
AI literacy
The knowledge and skills people need to use AI well and understand its risks. The EU AI Act asks organizations to support it.
Vendor due diligence
Checking a supplier's security, data handling, terms and track record before using its product.

Remember: Approved tools, clear data rules, allowed and banned uses, a person accountable for every output, disclosure, training, an owner and a review date.

Lesson 4: Open debates and careers in responsible AI

Goal: Present the main positions in the field's open debates fairly, and plan a route into a responsible AI role.

Some questions in this field are not settled by evidence, because they involve values and predictions. A professional's job is not to pick a team, but to state each position at its strongest (steelmanning), separate factual claims from value judgments, and say what evidence would change the picture. Three debates come up constantly.

DebateThe case for one sideThe case for the other side
How much regulation, and how soon?Harms are already documented; markets reward speed over safety; clear rules build trust, give everyone the same baseline and are easier to set before practices hardenRules can lock in today's assumptions about a fast-moving technology; compliance costs fall hardest on small firms and open projects; existing laws already cover many harms; heavy rules may push development elsewhere
Open or closed model weights?Open weights let independent researchers inspect and test models, support competition and privacy (models run locally), and avoid concentrating power in a few companiesOnce weights are released they cannot be recalled, and safeguards can be removed by fine-tuning, which matters most for the most capable models
Catastrophic risk or present harms?Some researchers and industry leaders warn that advanced AI could cause large-scale or even existential catastrophe, and a 2023 public statement called reducing extinction risk from AI a global priorityOthers say such claims are speculative, can distract from documented harms such as discrimination, surveillance and labour exploitation, and can be used to justify concentrating control

Worked example: a balanced briefing in five moves

  1. Question from an invented city council: "Should our departments be allowed to run open-weight AI models on city computers?"
  2. Facts: open-weight models can run on the city's own servers, so residents' data never leaves the city; their built-in safeguards can be weakened by anyone who fine-tunes them; city staff would have to patch and monitor them, as with any software.
  3. Values in tension: residents' privacy and the city's independence from vendors, against the risk of misuse and the cost of running the models safely.
  4. Each side at its strongest: supporters say local models are the most private option and avoid lock-in to one supplier; critics say the city lacks staff to evaluate and secure them, and a vendor contract gives clearer accountability.
  5. What would change the picture: an independent evaluation of the specific models, a staffing plan, and the results of a small pilot. Options, not a verdict: allow approved models for low-risk tasks with a review, pilot first, or wait.

Separate facts from values, steelman each side, name the evidence that would settle the disagreement, and give decision-makers real options.

Many positions sit in between: risk-based rules that tighten with capability or stakes, regulatory sandboxes, staged release of open models after testing, and the observation that many safeguards (evaluations, incident reporting, transparency, independent audits) serve both present-harm and catastrophic-risk concerns. Governments have also chosen differently: the EU legislated broadly, the US federal government has emphasized innovation while states legislate, and the 2024 US Commerce Department review of open-weight models recommended monitoring rather than restricting them for now.

RoleWhat the work involvesSkills this course builds
AI governance or compliance analystInventories AI systems, runs assessments, maps laws to controls, prepares auditsFrameworks, EU AI Act tiers, risk registers, documentation
Responsible AI program managerRuns the organization's AI policy, review board and trainingPolicy writing, oversight design, incident response
AI policy analystResearches and drafts law and policy for government, think tanks or companiesReading laws carefully, balanced argument, "as of" accuracy
Trust and safety specialistWrites and enforces platform rules, investigates abuse, handles appealsSynthetic media, provenance, precision and recall, transparency
AI auditor or assurance specialistTests systems and controls independently, including bias auditsFairness metrics by hand, model cards, evidence standards
AI red teamer or evaluatorDesigns tests that find failures before releasePrompt-injection triage, attack success rates, honest reporting
AI literacy educator or trainerTeaches staff, students or the public to use AI wellPlain-language explanation of every topic here

Credentials to aim for next, as of November 2026 (check each body's current requirements before enrolling): the IAPP Artificial Intelligence Governance Professional (AIGP), launched in April 2024 and the best-known AI governance credential, with its body of knowledge updated in 2026; IAPP privacy credentials such as CIPP/E, CIPP/US and CIPM for privacy-heavy roles; ISACA's Advanced in AI Audit (AAIA, 2025), Advanced in AI Security Management (AAISM, 2025) and Advanced in AI Risk (AAIR, April 2026), each of which requires an existing qualifying certification such as CISA, CISM or CRISC; and ISO/IEC 42001 lead implementer or lead auditor training from accredited training providers. Portfolio evidence matters as much: this course's final project, a model card for an open model, a bias audit of a public dataset, or an AI policy written for a real local organization.

Steelmanning
Stating an opposing view in its strongest form before responding to it.
Open-weight model
A model whose trained parameters are published so anyone can download, run and modify it.
Frontier model
One of the most capable AI models at a given time, often trained with the largest amounts of computing power.
Existential risk (from AI)
The possibility that advanced AI could cause human extinction or permanently and drastically curtail humanity's future.
AIGP
The IAPP's Artificial Intelligence Governance Professional certification.
Regulatory sandbox
A supervised setting where organizations can test new technology with regulators before full rules apply.

Remember: On open questions, steelman every side and separate facts from values. For your career: a credential such as the AIGP plus portfolio evidence.

Practise it in Responsible AI & AI Safety

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: AI risk register, Incident report writer and AI use-policy drafter.

Every lesson is free: no ads, no lives and no paywall.