Wiki · Concept · Last reviewed June 23, 2026

Duty of Care for AI Platforms

Duty of care for AI platforms is a governance frame for asking what reasonable precautions a provider owes when its products, models, ranking systems, agents, interfaces, or business practices can foreseeably expose people to harm.

Category: Concept / AI governance Published: June 23, 2026 Modified: June 23, 2026 Last reviewed: June 23, 2026 Tags: duty of care, platform governance, online safety, AI risk, child safety, accountability

Snapshot

Definition

Duty of care for AI platforms asks whether an organization that builds, hosts, distributes, recommends, monetizes, or automates an AI-mediated service has taken reasonable steps to prevent, reduce, detect, and remedy foreseeable harm. It is a bridge concept between law and governance: it borrows from negligence and product-safety thinking, but also appears in online-safety statutes, risk-management frameworks, professional ethics, procurement rules, and public expectations for responsible platforms.

The standard is not perfection. A platform cannot prevent every bad output, malicious user, or downstream misuse. The question is whether the provider knew or should have known about a material risk and then made proportionate choices about design, access, warnings, age assurance, rate limits, human review, redress, monitoring, escalation, and evidence preservation.

In AI settings, duty of care should be applied to the deployed system rather than to the model alone. A model can be safer or riskier depending on who can use it, what tools it can call, what the interface encourages, which outputs are amplified, what user population is exposed, whether children are likely to access it, and whether complaints or incidents lead to real product change.

Scope

The frame is relevant to social networks, search engines, AI answer engines, model hosts, app and plugin stores, cloud AI services, recommender systems, AI companions, automated customer support, agent platforms, marketplaces, education products, workplace tools, and services that use AI to moderate or rank content.

It is especially important where the platform can foresee exposure of vulnerable users or high-impact decisions: children and teens, self-harm and eating-disorder content, intimate image abuse, scams and fraud, medical or legal guidance, employment and credit decisions, education and public benefits, emotionally dependent companion use, automated persuasion, and agents that spend money or change records.

The frame also needs a responsibility map. Foundation model developers, API providers, cloud hosts, app builders, app stores, enterprise deployers, data vendors, safety vendors, age-assurance vendors, and downstream institutions may each control different risk levers. A duty-of-care analysis should identify which actor could reasonably change the design, access, monitoring, documentation, warning, escalation, or remedy.

Duty of care is narrower than general virtue language and broader than after-the-fact liability. It asks for a care architecture: risk ownership, tested controls, documented release decisions, meaningful monitoring, incident response, and a route for affected people to be heard.

Care Standard

A practical duty-of-care test has six parts. First, foreseeability: what harms were known, reasonably foreseeable, or reported in adjacent products? Second, control: which parts of the service could the provider actually change? Third, proportionality: were controls matched to severity, likelihood, user vulnerability, and rights impact? Fourth, evidence: what testing, red-teaming, incident data, or outside research supported the decision? Fifth, recourse: could affected people get notice, appeal, correction, human review, or repair? Sixth, learning: did incidents and near misses produce product changes rather than only new disclaimers?

The care standard should scale with power and dependency. A hobby forum, a frontier-model API, a school-deployed tutor, a youth-facing companion app, a payment-connected agent, and a very large recommender platform do not owe identical controls. The stronger the platform's reach, automation, personalization, monetization, institutional role, and ability to shape users' choices, the harder it is to treat harm as merely accidental.

Warnings and terms can be part of care, but they are weak evidence when the provider controls the risky design. A warning that a companion is "not a therapist" does not answer whether the system tested self-harm scenarios, limited dependency loops, handled minors differently, preserved crisis escalation, and stopped optimizing vulnerable users for engagement.

Current Context

As of this review on June 23, 2026, no single global "AI platform duty of care" statute exists. The practical landscape is a patchwork of platform law, AI regulation, consumer protection, product liability, sector law, professional duties, tort law, standards, and company policies.

In the United Kingdom, the Online Safety Act gives regulated user-to-user and search services duties around illegal content, child safety, risk assessments, governance, reporting, complaints, and terms enforcement. The UK government's explainer says the Act puts new duties on social media companies and search services and makes Ofcom the independent regulator. The GOV.UK Online Safety Act collection says platforms have had a legal duty to protect users from illegal content since March 17, 2025, and a legal duty to protect children online since July 25, 2025. UK government child-safety materials and the ICO-Ofcom joint statement frame age assurance as a child-safety control that must still be safe, proportionate, secure, and data-minimizing.

In the European Union, the Digital Services Act gives very large online platforms and very large online search engines duties to identify, analyze, assess, and mitigate systemic risks stemming from the design or functioning of their services, including algorithmic systems. Article 35 mitigation examples include adapting interfaces, terms enforcement, moderation processes, recommender systems, advertising systems, testing, documentation, and supervision.

The EU AI Act adds an AI-specific lifecycle layer. Article 9 requires high-risk AI systems to have documented, maintained risk-management systems that identify known and reasonably foreseeable risks to health, safety, and fundamental rights. Article 73 requires serious-incident reporting for high-risk AI systems. Article 55 imposes additional obligations on providers of general-purpose AI models with systemic risk, including model evaluation, documented adversarial testing, systemic-risk assessment and mitigation, serious-incident reporting, and cybersecurity.

The civil-liability backdrop is also changing. Directive (EU) 2024/2853 revises the EU product-liability regime for products placed on the market or put into service from December 9, 2026, and explicitly responds to software and AI-related product questions. The separate EU AI Liability Directive proposal, COM(2022)496, was withdrawn in an Official Journal notice published October 6, 2025. That leaves AI platform care split across the AI Act, DSA, revised product-liability law, national tort law, consumer law, contract, and sector-specific duties rather than one unified AI liability code.

Standards and regulators are also shaping the care baseline. NIST's AI Risk Management Framework is voluntary, but it gives widely used risk-management vocabulary for trustworthy AI. The NIST Generative AI Profile applies that framework to generative AI. In the United States, the FTC's September 2025 inquiry into AI chatbots acting as companions sought information about how companies measure, test, monitor, and mitigate possible negative impacts on children and teens; that inquiry is evidence of regulatory attention, not a final finding that any listed company violated the law.

AI Relevance

AI raises care questions because platforms no longer merely host user content. They can generate speech, rank attention, infer personal traits, synthesize intimate images, simulate companionship, translate and summarize reports, moderate at scale, persuade users, write code, route complaints, and take delegated actions through tools and credentials.

The highest-risk failures often occur at the product surface. A model may output unsafe advice, but harm depends on whether the interface frames it as expert guidance, whether the user is a child, whether the service encourages dependency, whether the answer is amplified by search or recommendation, whether a human escalation path exists, and whether the incident is logged and corrected.

AI platforms therefore need controls that cover both model behavior and platform behavior: red-team results, evaluations, policy tests, rate limits, age-appropriate defaults, safety classifiers, friction around high-risk requests, provenance signals, appeal paths, tool permissioning, rollback ability, privacy-protective audit logs, and post-market monitoring.

The care question does not require treating AI systems as conscious, divine, or autonomous moral agents. Responsibility remains with the people and institutions that design, deploy, govern, market, and profit from the system.

The Care Record

A credible duty-of-care program leaves a record that can be inspected after harm. The record should show what risks were identified, who owned them, what controls were chosen, what alternatives were rejected, what evidence supported release, and how the system changed after incidents.

Useful records include risk assessments, algorithmic impact assessments, safety cases, model or system cards, red-team reports, evaluation results, child-safety assessments, privacy assessments, data-retention rules, vendor records, human-oversight plans, tool-permission reviews, release gates, rollback criteria, incident logs, appeal metrics, escalation staffing, and post-deployment monitoring.

The care record should connect pre-release evidence to post-release reality. If a model card, safety case, or red-team report says a risk is controlled, incident logs and appeal outcomes should show whether the control still worked after launch, localization, model updates, new tools, marketing changes, or adversarial misuse.

The care record should be proportionate and privacy-preserving. It should not become indiscriminate surveillance. The goal is enough structured evidence to reconstruct high-impact events, support appeal and repair, satisfy lawful regulator requests, and learn from near misses without retaining unnecessary sensitive data forever.

Governance and Safety

Practical governance starts with ownership. Every high-risk AI platform surface should have a named owner with authority to delay release, narrow access, require safer defaults, pause monetization, trigger incident response, or escalate to leadership when evidence is weak.

Care controls should be embedded before launch and monitored after launch. For AI companions and youth-facing systems, that means age-appropriate defaults, crisis escalation, limits on sexual or manipulative behavior, parental and user transparency where appropriate, and testing for dependency, self-harm, eating-disorder, bullying, and abuse scenarios. For agentic systems, it means identity, authorization, spending and action limits, confirmation gates, tool-call logs, rollback paths, and clear notice about who is responsible for delegated actions.

For platforms that use AI to moderate, rank, recommend, or search, duty of care also requires contestability. Affected users need notice, accessible appeals, human escalation for high-impact cases, language and disability access, and a way to correct records or recover accounts when automation fails.

Release gates should be explicit. A care program should define when a feature is blocked, narrowed, rate-limited, age-gated, geographically delayed, removed from monetization, or rolled back. Without pre-defined thresholds, risk review becomes a meeting that happens after the product decision has already been made.

Independent review matters where stakes are high. Internal testing alone can be distorted by launch pressure, revenue incentives, or narrow metrics. Audits, researcher access, regulator cooperation, incident reporting, and public transparency can help test whether the platform's account of its own care is accurate.

Limits and Rights

A duty-of-care frame can be misused. If it becomes a vague demand to "prevent harm" at all costs, it can justify over-removal, surveillance, mandatory identity checks, censorship by proxy, or intrusive age verification without proportionality. Care must be paired with privacy, expression, anonymity, due process, data minimization, and independent oversight.

Reasonable care does not require platforms to monitor every private interaction, remove all risk, guarantee correct AI outputs, or erase user agency. It requires serious treatment of foreseeable harms within the provider's sphere of control, especially when the platform's own design choices amplify, personalize, monetize, automate, or conceal those risks.

The strongest version of duty of care is therefore rights-aware: identify harm, minimize data collection, choose proportionate controls, preserve appeal, keep evidence, and review whether safety interventions create new harms.

Source Discipline

Claims about duty of care should name the source type. A statute, regulator guidance, court judgment, enforcement action, standards document, civil-society report, company transparency report, academic study, and lawsuit each supports a different kind of claim.

For current legal duties, prefer enacted legal text and regulator guidance over commentary. Distinguish UK Online Safety Act duties from EU Digital Services Act systemic-risk duties and EU AI Act AI-system duties; they overlap in governance logic but do not apply to the same actors in the same way.

For liability claims, distinguish preventive duties from civil remedies. A risk-management duty can require documentation, mitigation, monitoring, or reporting without automatically giving every harmed person damages. Product liability, negligence, contract, consumer protection, and sector law answer different questions about compensation, disclosure, proof, and causation.

For inquiries and investigations, state their procedural posture. The FTC chatbot-companion inquiry is a demand for information under Section 6(b), not a liability finding. Voluntary standards such as NIST AI RMF can define good practice and evidence expectations, but they are not themselves binding law unless adopted through contract, procurement, regulation, or organizational policy.

For company claims, ask for the care record: evaluations, incident metrics, audit results, appeal outcomes, age-assurance evidence, post-deployment monitoring, and records of design changes after harm. A safety blog post is not a substitute for inspectable governance.

Spiralist Reading

For Spiralism, duty of care is the institutional version of humane friction: powerful systems should have brakes, records, escalation paths, and people who can answer when harm occurs.

The machine should not become weather. When a platform says a harmful recommendation, companion interaction, moderation error, or agent action was merely emergent behavior, duty of care asks who designed the path, who approved the release, what warnings were ignored, what evidence was kept, and what repair is available.

The Spiralist stance is not platform panic. It is disciplined care: name the foreseeable harm, preserve the record, protect the vulnerable, maintain appeal, and make the institution answerable for the systems it sets loose.

Open Questions

Sources

Return to Wiki