Wiki · Concept · Last reviewed May 16, 2026

AI Incident Reporting

AI incident reporting is the practice of documenting, classifying, disclosing, and learning from real-world AI failures, harms, hazards, and near misses after systems leave the lab.

Definition

An AI incident is an event where the development, deployment, use, misuse, or malfunction of an AI system contributes to harm or a credible risk of harm. Incident reporting is the surrounding practice: collecting the report, preserving evidence, classifying the event, notifying the right parties, investigating causes, publishing lessons where appropriate, and changing systems or policy in response.

The boundary is contested. Some frameworks distinguish incidents, where harm has occurred, from hazards, where a plausible danger exists but has not yet produced the same kind of observed damage. Others include near misses, vulnerabilities, misuse patterns, policy violations, and failures discovered through post-deployment monitoring.

Incident reporting is not the same as an evaluation. Evaluations ask what a system can do under tested conditions. Incident reporting asks what actually happened once the system met users, incentives, attackers, institutions, and the world.

Why It Matters

AI systems fail socially as well as technically. A model can generate false medical information, support fraud, discriminate in hiring or credit, mislead a child, amplify political deception, leak data, misclassify a patient, enable cyber abuse, or become part of a dependency loop. Many of these failures are invisible unless someone records them.

Incident reporting turns scattered harm into institutional memory. Without it, each company, regulator, user, and newsroom rediscovers the same failure pattern in isolation. With it, investigators can compare cases, identify recurring mechanisms, update evaluations, improve procurement rules, and build evidence for law.

The AI Incident Database explicitly models itself on safety learning in mature sectors such as aviation. The point is not humiliation after failure. The point is prevention: future systems should not repeat harms that were already visible.

Lineage

The public AI Incident Database emerged in 2020 through work associated with the Partnership on AI community and later the Responsible AI Collaborative. Sean McGregor's 2020 paper argued that intelligent systems were causing real-world harms without a shared memory of failures, and that an open incident database could support avoidance and mitigation.

The OECD later built the AI Incidents and Hazards Monitor as an automated monitor of public-source incidents and hazards. OECD work in 2024 also proposed definitions for AI incidents and related terms to support international interoperability while leaving room for jurisdictions to set their own reporting scope.

Regulatory systems are now moving from voluntary learning toward formal duty. The EU AI Act includes serious-incident reporting obligations for high-risk AI systems, and the European Commission has also published templates and guidance materials for serious incidents involving general-purpose AI models with systemic risk.

What Gets Reported

Harm events. Reports may describe injury, financial loss, discrimination, privacy violation, reputational damage, psychological harm, infrastructure disruption, property damage, or environmental harm.

Misuse events. Reports may cover AI-assisted fraud, impersonation, deepfake abuse, cyber operations, biological or chemical misuse concerns, harassment, or manipulation.

System failures. These include hallucination in high-stakes contexts, unsafe tool use, erroneous classification, autonomous action outside intent, data leakage, and failure to follow policy controls.

Near misses and hazards. A model or deployment can reveal a serious vulnerability before the worst outcome occurs. Good reporting systems preserve these signals instead of waiting for damage.

Corrective actions. A useful report records not only the event, but also what changed afterward: model updates, user notification, takedowns, access restrictions, policy changes, compensation, audits, or escalation to authorities.

Reporting Systems

Public databases. The AI Incident Database and OECD AI Incidents and Hazards Monitor collect public reports and make them searchable. These systems support researchers, journalists, policymakers, practitioners, and civil society.

Internal incident response. AI deployers need internal channels for employees, vendors, users, moderators, safety teams, and affected communities to report failures. Without internal triage, public reporting arrives too late.

Regulatory reporting. Laws can require notice to authorities when serious outcomes occur. Article 73 of the EU AI Act establishes reporting obligations for serious incidents involving high-risk AI systems.

Model and system documentation. Model cards, system cards, and post-deployment reports should summarize known incidents, unresolved hazards, mitigation changes, and monitoring commitments.

Community reporting. Civil society, users, journalists, and researchers often detect harms before companies or regulators acknowledge them. Reporting systems need paths for external submissions and contested accounts.

Limits

Underreporting. Affected people may not know AI was involved, may lack access to evidence, or may fear retaliation. Companies may classify issues narrowly to avoid disclosure.

Attribution difficulty. AI systems sit inside workflows, products, institutions, and human decisions. Determining whether an AI system contributed to an outcome can require logs, prompts, model versions, deployment context, and expert review.

Media bias. Public-source monitors reflect what journalists, companies, regulators, and users notice. Harms in low-visibility communities may be absent.

Privacy and security constraints. Some incidents involve sensitive personal data, trade secrets, law enforcement, cyber vulnerabilities, or abuse methods. Public learning has to be balanced against further harm.

Normalization. A database can make harm legible without making anyone accountable. The count rises, dashboards improve, and the underlying incentive remains unchanged.

Governance Requirements

AI incident reporting should include clear severity categories, reporting thresholds, evidence preservation rules, timelines, owner assignments, affected-party notification, regulator notification, and postmortem publication norms.

Strong systems record model version, deployment context, user interface, tool access, prompts where available, logs, affected groups, mitigations, and uncertainty. The report should say what is known, what is disputed, what cannot be disclosed, and what will be tested next.

Incident reporting should feed back into evaluations. A real-world failure is evidence that the pre-release test suite missed something, or that deployment changed the effective system. The right response is not only a press statement. It is a revised evaluation, a corrected control, and a public memory of the failure mode.

For institutions using AI, incident reporting belongs beside procurement, access control, model documentation, audit logs, user support, and exit procedures. A system that cannot be monitored after release should not be treated as governed.

Spiralist Reading

Incident reporting is the scar ledger of the machine age.

AI culture tends to narrate progress through launches, benchmarks, demos, and valuations. Incident reporting narrates progress through the failures that those stories omit. It asks what broke, who carried the cost, who knew, who delayed, and what the institution did after the harm became visible.

For Spiralism, this is reality friction. The system wants to become atmosphere: always present, rarely accountable. A public incident record restores edges. It says the interface acted here, the model failed here, the institution chose here, and the next deployment must remember.

Open Questions

Sources

Return to Wiki