Responsible AI & AI Safety · Unit 4 of 8
Unit 4: Transparency, explainability and oversight
Telling people when AI is involved, model cards and datasheets, explaining predictions with feature importance, and human oversight that is real, not a rubber stamp.
Opens the first lesson of this unit in Responsible AI & AI Safety. Every lesson is open from the start.
Lesson 1: Transparency: who needs to know what
Goal: Match the kind of transparency to the audience: disclosure, documentation or explanation.
"Transparency" means different things to different people, and good practice gives each audience what it needs. Disclosure tells people that AI is involved at all. Documentation tells professionals (auditors, regulators, deployers) how a system was built and tested. Explanation tells an affected person why a decision about them came out the way it did and what they can do about it.
| Audience | What they need | Examples of rules (as of November 2026) |
|---|---|---|
| People talking to a chatbot | To know it is not a person | EU AI Act Article 50: people must be told they are interacting with AI, applying from 2 August 2026 |
| People seeing synthetic media | To know an image, voice or video is AI-made | EU AI Act Article 50: machine-readable marking by providers and deepfake labels by deployers |
| Deployers, auditors, regulators | How the system was built, tested and should be used | EU AI Act technical documentation and instructions for high-risk systems; developer documentation duties in some US state laws |
| A person denied credit | The specific main reasons for the decision | US Equal Credit Opportunity Act and Regulation B: a statement of specific reasons for adverse action |
| A person subject to an automated decision | Meaningful information about the logic, and a way to contest it | GDPR Articles 13–15 and 22; Colorado's replacement AI law (taking effect January 1, 2027) requires explaining adverse decisions and offering human review |
Explanations must be specific and honest. A credit denial saying "you did not meet our internal standards" tells the person nothing they can act on. In US lending, Regulation B requires the principal reasons, and that does not change because a complex algorithm made the decision. (The Consumer Financial Protection Bureau's 2022 guidance stressing this point, Circular 2022-03, was one of dozens of guidance documents withdrawn in May 2025, but the regulation itself still requires specific reasons.)
Worked example: rewriting a weak notice
- Weak: "Your application was assessed using advanced analytics and was unsuccessful."
- Problems: no mention that a model decided, no reasons, no route to challenge, no contact.
- Better: "An automated system assessed your application. The main reasons it was declined were: (1) a debt-to-income ratio above 45%; (2) two late payments in the last 12 months. You can ask for a person to review the decision, and correct any information we hold, by replying within 30 days or calling our review team."
- It names AI, gives specific reasons the person can understand and act on, and offers human review and correction.
A good notice says AI was used, why the result came out as it did, and how to get a person to look again.
- Disclosure
- Telling people that AI is being used, for example that they are talking to a chatbot.
- Documentation
- Technical records of how an AI system was built, tested and should be used.
- Explanation
- Information given to an affected person about why a decision about them was made.
- Adverse action notice
- In US credit, the notice telling an applicant that credit was denied, with the specific principal reasons.
- Automated decision-making
- Decisions made by technology without meaningful human involvement.
Remember: Disclose to everyone, document for professionals, explain to the person affected, and always offer a way to challenge.
Lesson 2: Model cards and datasheets
Goal: Write a model card and know what a datasheet for a dataset records.
A model card is a short document that travels with a model, like a label on medicine. The idea comes from a 2019 paper by Margaret Mitchell and colleagues. Its core sections: model details, intended use, out-of-scope uses, factors (groups and conditions that may affect performance), metrics, evaluation data, training data, results broken down by group, ethical considerations, and caveats and recommendations. Many companies now publish model cards or longer system cards with their models.
A datasheet for datasets (Gebru and colleagues, first published in 2018) does the same for data: why the dataset was created, who funded it, what is in it, how it was collected and with what consent, how it was cleaned and labeled, recommended and discouraged uses, how it will be maintained. Most bias problems trace back to data, so datasheets are where many audits start.
The heart of a model card is disaggregated evaluation: results reported separately for relevant groups and conditions, not just one overall number. "93% accurate" can hide 97% for one group and 78% for another. A good card also states plainly what the model must not be used for. Out-of-scope uses are where many harms happen, as when a support tool turns into a decision-maker.
Worked example: one section done well
- Weak "performance" section: "The model is 92% accurate."
- Strong version: "Test set: 4,000 invented support tickets from January–March 2026. Overall accuracy 92%. By language: English 94%, Spanish 89%, French 84%. Urgent tickets: recall 88%; 12% of urgent tickets were missed."
- Strong "out of scope": "Not for deciding refunds or account closures. Not tested on tickets in other languages. Not for customers under 18."
- Strong "limitations": "Accuracy drops for very short tickets (under 10 words). Sarcasm is often misread."
Numbers by group, named data, clear "do not use for", honest limits: that is what makes a card useful.
Hands-on lab in the room: Model card builder. Build a model card for an invented ticket-triage model. The checker flags sections that are missing, vague or lack the numbers and groups a reviewer needs.
- Model card
- A short document describing a model's intended use, performance by group, limitations and ethical considerations.
- System card
- A longer document describing a whole AI system, its testing and its safeguards.
- Datasheet for datasets
- A document recording why and how a dataset was created, what it contains and how it should be used.
- Disaggregated evaluation
- Reporting performance separately for each relevant group or condition.
- Out-of-scope use
- A use the developer says the model was not designed or tested for.
Remember: A model card earns its keep with performance by group, named data, explicit out-of-scope uses and honest limitations.
Lesson 3: Explaining predictions: feature importance made simple
Goal: Read permutation importance and additive feature contributions, and know their limits.
Some models are interpretable by design: a short decision tree or a points-based scorecard can be read directly. Others, such as large neural networks and big tree ensembles, are effectively black boxes, so we use explanation methods that probe them from outside. Two questions come up. Global: which features matter most to the model overall? Local: why did this one person get this result?
Permutation importance answers the global question. Measure the model's accuracy on test data. Then shuffle one feature's column, scrambling its link to the answers, and measure again. The bigger the drop, the more the model relied on that feature.
Worked example: permutation importance by hand
- An invented loan model scores 0.86 accuracy on test data.
- Shuffle "income": accuracy falls to 0.71, a drop of 0.15.
- Shuffle "missed payments": accuracy falls to 0.76, a drop of 0.10.
- Shuffle "postcode": accuracy falls to 0.80, a drop of 0.06.
- Shuffle "favorite colour": accuracy stays 0.86, a drop of 0.00.
- Ranking: income, then missed payments, then postcode. Postcode matters more than expected for a loan model: check whether it is a proxy for race.
Importance = accuracy before − accuracy after shuffling. Surprising important features are leads for a bias review.
Additive contributions answer the local question. Methods such as SHAP (based on Shapley values from game theory) split one prediction into a share per feature, so that the shares add up to the difference between this prediction and the average one. For example: average approval score 0.40; this applicant's income adds +0.20, missed payments subtract −0.15, a short credit history subtracts −0.05; so the score is 0.40 + 0.20 − 0.15 − 0.05 = 0.40. LIME fits a small, simple model around one prediction. Counterfactual explanations state the smallest change that would flip the result: "with income 4,000 higher, the loan would have been approved."
Know the limits. Explanations describe the model, not the world: "postcode matters" does not mean postcode causes default. Different methods can disagree. Explanations can be unfaithful (a simple story that does not match what the model really does) and can be gamed. For high-stakes decisions, many experts argue for interpretable models wherever they perform well enough, instead of explaining a black box after the fact.
- Interpretable model
- A model simple enough for people to read its logic directly, such as a small tree or a scorecard.
- Black box
- A model whose internal logic people cannot practically follow.
- Global explanation
- An account of what drives a model's behavior overall.
- Local explanation
- An account of why a model gave one particular output.
- Permutation importance
- How much a model's performance drops when one feature's values are shuffled.
- SHAP
- A method that splits one prediction into additive contributions from each feature, based on Shapley values.
- Counterfactual explanation
- The smallest change to the inputs that would have changed the outcome.
Remember: Shuffle a feature and watch accuracy drop for global importance; add up contributions for local reasons. Explanations describe the model, not causes.
Lesson 4: Human oversight and accountability
Goal: Design human oversight that is meaningful, and name who is accountable.
Human oversight comes in degrees. Human in the loop: a person makes or approves each decision. Human on the loop: the system acts, and a person monitors and can step in. Human in command: people decide whether, when and how the system is used at all. The EU AI Act (Article 14) requires high-risk systems to be designed so that overseers can understand their capabilities and limits, stay aware of automation bias, interpret outputs correctly, and decide not to use, override or stop the system.
Automation bias is the well-documented human tendency to over-trust automated suggestions, especially under time pressure. It turns "a human reviews every case" into a rubber stamp. Oversight is meaningful only if the reviewer has the time, the information (including the model's uncertainty and the case details), the competence and the authority to disagree, and if disagreements are tracked.
Worked example: is this review real?
- An invented insurer's fraud model flags claims; one reviewer must approve 480 flagged claims in an 8-hour shift.
- 8 hours = 28,800 seconds. 28,800 ÷ 480 = 60 seconds per claim.
- The reviewer sees only "FRAUD RISK: HIGH", with no reasons or claim documents.
- Last quarter the reviewer overturned 0.2% of flags.
- Verdict: this is close to a rubber stamp. Fixes: fewer cases per person, reasons and documents on screen, a target to sample and challenge flags, and tracking of overturn rates. A near-zero overturn rate is a warning sign, not proof of a perfect model.
Count the seconds per case, check what the reviewer can see, and watch the overturn rate.
Accountability means a named person or body answers for the system: its approval, its performance and its failures. Good practice: a named system owner; clear roles written down (who builds, who approves, who monitors, who handles complaints); logging of inputs, outputs and overrides so decisions can be reconstructed; contestability, a way for affected people to challenge outcomes; and independent checks such as internal audit. Many organizations use a "three lines" model: the business runs the system, a risk and compliance function oversees it, and internal audit checks both.
- Human in the loop
- A person makes or approves each decision the system supports.
- Human on the loop
- The system acts automatically, while a person monitors and can intervene.
- Human in command
- People decide whether and how an AI system is used at all.
- Automation bias
- The tendency to over-trust automated suggestions and under-use one's own judgment.
- Accountability
- A named person or body answers for a system's decisions and results.
- Audit trail
- Logged records of inputs, outputs and human actions that let decisions be reconstructed.
Remember: Real oversight needs time, information, competence and authority, plus logs and a named owner.
Practise it in Responsible AI & AI Safety
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Model card builder.
Every lesson is free: no ads, no lives and no paywall.