Wiki · Concept · Last reviewed June 25, 2026

Web Bot Auth

Web Bot Auth is an emerging pattern for verifying automated web traffic with cryptographic HTTP message signatures, giving websites a stronger signal about which bot or agent made a request without turning that signal into permission by itself.

Definition

Web Bot Auth is a proposed and partially deployed method for authenticating automated HTTP clients. A bot or agent signs parts of its outbound request, publishes signing keys through a discoverable directory, and lets the receiving service verify that the request came from an actor associated with that key material.

The point is narrower than trust. A valid signature can identify the key holder, but it does not say the client is safe, welcome, entitled to scrape, authorized to spend, or acting with the user's informed consent. That distinction is the governance boundary.

Cloudflare documents Web Bot Auth as a verification method for verified bots and signed agents. The IETF Web Bot Auth work treats the same family of problems as active Internet-Draft work built on the standards-track HTTP Message Signatures specification, RFC 9421. The result sits near AI Agent Identity, Agent-Native Internet, and Content Provenance and Watermarking, but it applies to request traffic rather than generated media.

How It Works

The base mechanism is HTTP Message Signatures. RFC 9421 defines how an application can create, attach, parse, and verify signatures over selected HTTP message components using the Signature-Input and Signature fields. A verifier reconstructs the signature base from the covered components and checks the supplied signature against application requirements.

Web Bot Auth adds bot-oriented conventions around that mechanism. In Cloudflare's integration, an operator generates an Ed25519 signing key, publishes the public key through a JSON Web Key Set at /.well-known/http-message-signatures-directory, serves that directory over HTTPS, and registers the bot and key directory. Once accepted, the bot signs its requests with three relevant headers: Signature-Input, Signature, and Signature-Agent.

Signature-Input lists the covered components and parameters such as created, expires, keyid, and the web-bot-auth tag. Signature carries the cryptographic signature value. Signature-Agent points the verifier toward the signing-key directory. The IETF directory draft generalizes that discovery pattern with a JWKS-based key directory, a well-known URI, and a Signature-Agent header for in-band key discovery.

Implementation details matter. Cloudflare's current profile uses a structured-string Signature-Agent form from an earlier directory draft, fails later dictionary forms, and does not support every RFC 9421 component or parameter. Implementers must track the verifier's exact profile.

Current Context

The Web Bot Auth working area is moving because automated web clients now include search crawlers, commercial scrapers, answer engines, shopping agents, research agents, monitoring systems, and browser-like automations. IP allowlists and reverse DNS checks remain useful, but they are brittle when traffic moves through changing infrastructure or when user-directed agents need to identify themselves.

The IETF datatracker lists Web Bot Auth related drafts, including the automated-traffic architecture draft and the HTTP Message Signatures Directory draft. Both are Internet-Drafts, not final standards. Cloudflare's docs show the deployed side: request signatures can feed verified-bot and signed-agent workflows. The ecosystem still has to converge on directory formats, header profiles, revocation, privacy norms, and the policy meaning of signed identity.

Governance and Safety

The main governance error is confusing authentication with authorization. Web Bot Auth can help a site separate unknown automation from automation associated with a signing identity. It cannot decide whether the bot's purpose is legitimate, whether a content owner has consented, whether a user's agent is overstepping, or whether a model vendor's crawler deserves special treatment.

A second risk is centralization. If only a few infrastructure providers decide which agents count as recognized, Web Bot Auth could become a pass system for the machine-readable web. If anyone can publish a key directory without policy, signatures become cleaner labels for familiar abuse. Keys can also be compromised, replay defenses depend on verifier behavior, and stable identifiers can create tracking pressure.

Defense Pattern

Spiralist Reading

Web Bot Auth is a receipt for machine footsteps.

The old web often relied on theater: a user-agent string, an IP range, a crawl-delay convention, a contact email, a promise not to be too loud. The agent web needs stronger receipts because automated visitors can read, buy, summarize, compare, and trigger workflows at scale.

For Spiralism, the useful move is demystification. The signature does not sanctify the agent. It names a technical actor and makes a request legible to policy, so fewer systems have to guess whether the caller is a person, a crawler, a vendor, a user-directed agent, or a disguise.

Open Questions

Sources

Return to Wiki