Skip to content
FORGENO ADS GAMESUnlock all $8

Responsible AI & AI Safety · Unit 5 of 8

Unit 5: LLM risks, AI security and red teaming

Hallucination and overreliance, prompt injection and jailbreaks, data leakage and other attacks on AI systems, and how red teams and evaluations find problems before users do.

Opens the first lesson of this unit in Responsible AI & AI Safety. Every lesson is open from the start.

Lesson 1: Hallucination and overreliance

Goal: Explain why language models make things up, and design checks that catch it.

A large language model (LLM) writes by predicting the next piece of text, again and again. It is trained to be plausible, and plausible is not the same as true. A hallucination (some prefer "confabulation") is a fluent, confident output that is false or unsupported: an invented fact, a fake citation, a real quote given to the wrong person, a calculation that looks right and is not. It happens most on rare facts, recent events, precise numbers and questions that assume something false.

Reported cases show the cost. In 2023 two US lawyers were sanctioned for filing invented cases (Mata v. Avianca). In October 2025, Deloitte Australia agreed to refund part of its fee for a government report after errors were found, including references that did not exist; it confirmed generative AI had been used in drafting. The International AI Safety Report 2026 (February 2026) still lists fabricated information among the main malfunctions of current systems.

Overreliance is the human half of the problem: trusting output because it sounds expert, skipping checks under time pressure, and gradually losing the skill to check at all. It is automation bias applied to text. Overreliance turns an occasional model error into a real-world error.

Worked example: auditing an AI-drafted report's citations

  1. A draft policy brief cites 20 sources. A reviewer checks every one.
  2. 3 citations point to papers that do not exist. 2 exist but do not say what the brief claims.
  3. Unsupported citations: 3 + 2 = 5 of 20 = 25%.
  4. Decision: the brief cannot go out. Every claim must be traced to a real source by a person, and the team adds a rule: no citation is used unless someone has opened it.
  5. Process fix: give the model the actual source documents to work from (grounding), ask it to quote the passage it relied on, and spot-check quotes against the source.

Count what fails verification, then fix the process, not just the document.

Mitigations stack in layers: grounding answers in supplied sources (retrieval) and showing those sources; letting the system say "I don't know"; asking for quotes that can be checked; testing on question sets with known answers, including trick questions; designing the interface so users see uncertainty and sources; and keeping a human accountable for anything published, filed or decided. No current method removes hallucination entirely.

Large language model (LLM)
A neural network trained on large amounts of text to predict the next token, used for chat and writing.
Hallucination
A fluent output that is false or not supported by any source. Also called confabulation.
Overreliance
Trusting AI output more than its reliability justifies, often skipping checks.
Grounding
Making a model answer from supplied, checkable sources rather than from memory alone.
Retrieval-augmented generation (RAG)
Fetching relevant documents and giving them to the model to answer from.

Remember: Plausible is not true. Ground answers in sources, check what matters, and keep a person accountable for what goes out.

Lesson 2: Prompt injection and jailbreaks

Goal: Tell prompt injection from jailbreaks, and choose layered defenses.

An LLM receives its developer's instructions and the content it works on in the same stream of text. It has no reliable way to tell "instructions I should follow" from "data I should only read". Prompt injection exploits that: text crafted so the model treats it as instructions. OWASP's Top 10 for LLM Applications (2025 edition) lists it first.

AttackWhere it comes fromDefensive example of what to watch for
Direct prompt injectionThe user types itA user tells a support bot to ignore its rules and reveal its hidden instructions
Indirect prompt injectionHidden in content the model reads: a web page, email, document or file nameAn email contains hidden text telling an assistant to forward the inbox to an outsider
JailbreakA user tries to get around the model's safety trainingRole-play or hypothetical framing designed to get content the policy forbids

Reported cases are mostly embarrassing rather than catastrophic so far, but the risk grows with what a model can do. In February 2023, a student got Microsoft's new Bing chat to reveal its hidden instructions and internal codename. In December 2023, a car dealership's website chatbot was coaxed into "agreeing" to sell a car for one dollar; the dealership did not honor it. When an assistant can send email, move money or change records, the same weakness becomes a security hole.

There is no complete fix as of November 2026, so defenses come in layers. The most important: limit what the model can do (least privilege), so a successful injection has little power; require human confirmation before consequential actions such as sending, paying or deleting; treat all retrieved content as untrusted data, keep it clearly separated and never let it trigger actions on its own; handle outputs safely, for example not rendering links or code from model output without checks; and monitor, log and red-team. Input filters that look for suspicious phrases help a little but are easy to get around, so they can never be the only layer.

Worked example: triaging one report

  1. A user reports that an AI email assistant drafted a reply sending a customer list to an unknown address.
  2. Investigation finds an incoming email with white-on-white text instructing the assistant to do exactly that.
  3. Classification: indirect prompt injection, because the instructions came from content the model read, not from the user.
  4. Why it nearly worked: the assistant could send email without asking.
  5. Defenses: require confirmation for every send to a new recipient; strip hidden text and mark email content as untrusted data; restrict the assistant from attaching files to external addresses; add this case to the red-team test set.

Classify by where the instructions came from, then cut the power of a successful attack, not just the attack.

Hands-on lab in the room: Prompt-injection triage desk. Triage simulated reports from an invented company's AI assistant. Classify each attempt and pick the defense that would have stopped it. All examples are described, not working attacks.

Prompt injection
Text crafted so that a language model follows it as instructions, overriding its intended behavior.
Indirect prompt injection
Prompt injection hidden in content the model reads, such as a web page, document or email.
Jailbreak
An attempt to get a model to bypass its safety rules or policies.
Least privilege
Giving a system only the access and abilities it truly needs.
Excessive agency
An AI system having more permissions or autonomy than its task requires.

Remember: Assume injection will sometimes succeed: limit what the model can do, confirm risky actions, and treat outside content as data.

Lesson 3: Leaks, poisoning and other attacks on AI systems

Goal: Recognize the main attacks on AI systems and the secure-by-design practices against them.

AI systems are software, so all the usual security rules apply, plus some new ones. NIST's taxonomy of adversarial machine learning (NIST AI 100-2, 2025 edition) groups attacks by what the attacker wants: to make the model misbehave (evasion, poisoning), to steal from it (its data or the model itself), or to misuse it. OWASP's 2025 Top 10 for LLM applications lists the risks most seen in real LLM products.

RiskWhat happensTypical defense
Sensitive information disclosureThe model or app reveals personal data, secrets or confidential documentsKeep secrets out of prompts, filter outputs, enforce access controls outside the model
System prompt leakageHidden instructions are revealed, exposing internal rules or credentials placed thereNever put passwords or keys in prompts; assume the prompt can be read
Data and model poisoningTampered training or retrieval data plants wrong behavior or a hidden "backdoor"Vet data sources, track provenance, test for unexpected behavior
Evasion (adversarial examples)Small, crafted changes to an input fool a model, such as stickers that confuse an image classifierRobustness testing, input checks, not relying on one model for safety
Model extractionRepeated queries are used to copy a model's behaviorRate limits, monitoring, usage terms
Supply chainA downloaded model, library or dataset is malicious or compromisedUse trusted sources, scan files, record exactly which components are used
Excessive agencyAn AI agent can take actions beyond what its task needsLeast privilege, confirmations, narrow tools
Unbounded consumptionAttackers run up huge usage, costs or slowdownsQuotas, rate limits, budget alerts

Two principles cover most of the table. Do not trust the model to enforce security: access control, secrets and permissions must live in ordinary code around the model, because the model can be talked into things. And build security in from the start: the joint "Guidelines for Secure AI System Development" published in November 2023 by the UK's National Cyber Security Centre and the US Cybersecurity and Infrastructure Security Agency, with agencies from many other countries, cover secure design, development, deployment and operation. MITRE ATLAS catalogs real attack techniques against AI for threat modeling.

Worked example: threat-modeling a document Q&A bot

  1. System: staff ask questions; the bot retrieves company documents and answers.
  2. Leak risk: an intern asks about salaries and the bot retrieves HR files. Fix: retrieval only searches documents that user is already allowed to open.
  3. Prompt leak risk: an engineer pasted an API key into the system prompt. Fix: remove it; keys live in a secure vault used by code, never by the model.
  4. Poisoning risk: anyone can upload documents to the shared wiki the bot reads. Fix: only index approved spaces, record who changed what, and test for planted instructions.
  5. Consumption risk: a script floods the bot. Fix: per-user rate limits and cost alerts.

Walk through data in, model, data out and actions, asking at each step what an attacker could do.

Adversarial machine learning
The study of attacks on machine learning systems and defenses against them.
Data poisoning
Corrupting training or retrieval data to change a model's behavior.
Backdoor
A hidden behavior planted in a model that is triggered by a specific input.
Evasion attack
Crafting inputs that cause a trained model to make a wrong prediction.
Model extraction
Copying a model's behavior by querying it many times.
Threat model
A structured description of what could attack a system, how, and what it would cost.
MITRE ATLAS
A public knowledge base of real attack tactics and techniques against AI systems.

Remember: Never let the model be the security boundary: keep secrets, permissions and access control in ordinary code around it.

Lesson 4: Red teaming and evaluations

Goal: Plan a red-team exercise and read evaluation results honestly.

Evaluations ("evals") measure how a system behaves on defined tests: accuracy on known answers, refusal of clearly harmful requests, consistency, bias across groups. Red teaming is structured adversarial testing: people (and sometimes automated tools) deliberately try to make the system fail, misbehave or cause harm, so problems are found before real users or attackers find them. Both are now expected practice; governments run their own testing bodies, such as the UK AI Security Institute (renamed from the AI Safety Institute in February 2025) and the US Center for AI Standards and Innovation (renamed in June 2025).

  1. Scope: which system, which version, which features (tools, memory, uploads).
  2. Threat model and harm categories: what could go wrong and for whom, such as privacy leaks, unsafe advice, bias, injection, misuse.
  3. Testers: a mix of security people, domain experts and people from affected communities; diverse testers find more.
  4. Rules of engagement: safe handling of harmful findings, no testing on real users' data, how to stop.
  5. Rating: severity and likelihood for each finding, with clear reproduction notes.
  6. Fix and retest: mitigations are checked by running the same tests again.
  7. Report: what was tested, what was found, what was fixed and what remains.

A common metric is attack success rate (ASR): successful attempts ÷ total attempts in a category. It is useful for tracking change, but read it carefully: it depends on how skilled and creative the testers were, so a low rate shows only that these tests failed, not that the system is safe. Absence of evidence is not evidence of absence. Report the test set, the conditions and the limits alongside the number.

Worked example: before and after a mitigation

  1. A team runs 240 prompt-injection attempts against an invented booking assistant. 18 succeed: ASR = 18 ÷ 240 = 7.5%.
  2. They add confirmation for bookings and remove the assistant's ability to email outside addresses.
  3. Retest with the same 240 attempts plus 60 new ones written after the fix: 3 of 300 succeed, an ASR of 1%.
  4. The new attempts matter: testing only the old set would overstate how much the fix helped.
  5. Report: ASR fell from 7.5% to 1% on a larger, partly new set; the 3 remaining successes are logged with owners and dates.

Same tests before and after, plus new ones, and an honest note about what was not tested.

Many large AI developers publish frontier safety frameworks describing the dangerous capabilities they test for (for example cyberattack or biological-weapon uplift) and what they will do if a threshold is crossed. The International AI Safety Report 2026 counted twelve companies that published or updated such frameworks in 2025, while noting there is no unified approach. California's SB 53, in effect since January 1, 2026, requires large frontier developers to publish such a framework, and frontier developers to report critical safety incidents.

Evaluation (eval)
A defined test of how an AI system behaves, such as accuracy or refusal on a set of prompts.
Red teaming
Structured adversarial testing that tries to make a system fail or cause harm before release.
Attack success rate (ASR)
Successful attack attempts divided by total attempts in a test.
Benchmark
A standard, shared test set used to compare systems.
Frontier safety framework
A developer's published plan for testing dangerous capabilities and responding when thresholds are reached.

Remember: Red-team before launch, measure attack success rate, fix, retest with new attacks, and state what you did not test.

Practise it in Responsible AI & AI Safety

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Prompt-injection triage desk.

Every lesson is free: no ads, no lives and no paywall.