Cloudflare splits AI bots into Search, Agent, Training

For its second "Content Independence Day" (July 1), Cloudflare replaced the one-click "Block AI bots" option with finer controls: AI bots are classified by three behaviors — Search (collecting/indexing content to answer queries later), Agent (acting in real time on a user's behalf, e.g. ChatGPT-User or browser-use agents) and Training (crawling content to train or fine-tune models). The new options are available to all customers, including the Free tier.

Key points

• New defaults from September 15, 2026 for domains onboarding: Training and Agent blocked by default on ad-displaying pages; Search stays allowed to preserve discoverability.
• Multi-purpose crawlers (Googlebot, Applebot, BingBot — both Search and Training) will be enforced by the most restrictive rule: customers blocking Training also block them, unless the site owner opts out in Security settings.
• BotBase launches for Enterprise Bot Management: a searchable database of all known bots/agents with behavior classifications and detection IDs.
• A new use signal extends Content Signals in robots.txt (immediate/reference/full) to express how much a bot may store and reshare; abusers lose Verified status.
• A "transitive trust" proposal uses the Forwarded header (RFC 7239) to identify operators behind agents running through intermediary platforms.

Cloudflare also announced Attribution Business Insights and its agentic-Internet bot report the same day.