AI Bots Insights Report — Edition #5, July 2026

How AI bots access publisher content today, and which ones are getting blocked

Every month, Content Bridge analyses real bot traffic across our DACH publisher network to give the industry a clear picture of what's happening beneath the surface. Edition #5 adds a new dimension to that picture: the Crawler Blocking Index, showing which bots publishers are actively trying to keep out.

The blocking paradox

Nearly 70% of the top 120 German publishers now block CCBot and GPTBot via robots.txt. ClaudeBot, Google-Extended and Applebot-Extended each sit above 55%. The message from publishers is unambiguous: they want control over how AI companies access their content.

Yet the blocking is largely symbolic. robots.txt is a polite request, not a technical barrier. Bots that ignore it face no immediate consequence, and many do. Our traffic data shows that AI-related bots still account for around 24% of all bot traffic hitting publisher properties, a figure that has remained stable over recent months despite the widespread adoption of blocking rules.

robots.txt vs. llms.txt: a gap in tooling

The numbers paint a stark contrast. 95.9% of Germany's top publishers have a robots.txt file in place. Only 8.1% have adopted llms.txt, the newer protocol designed specifically for communicating preferences to AI crawlers and LLM agents.

What the traffic data shows

AI Search and Answer Engines remain the largest AI-related category at 15.1% of total bot traffic, followed by AI Training and Data Acquisition at 6.1%. When you combine all AI-related categories, they account for roughly one in four bot requests.

Amazonbot leads individual AI bots by request volume, followed by Applebot and Spider. The geographic origin of AI bot traffic is heavily concentrated in the United States, which generates the vast majority of AI-related requests, with Singapore as a distant second.

New: the Crawler Blocking Index

This edition introduces the Crawler Blocking Index, measuring the share of publishers that block each bot via robots.txt at the full-site level. The top 10 most blocked bots are all AI-related, with CCBot (69.9%) and GPTBot (69.1%) leading the list. Bytespider follows at 65%, ClaudeBot at 63.4%, and Google-Extended at 56.1%.

The pattern shows that publishers are using the tools they have to push back against AI crawling. But those tools are limited in scope and enforcement. The gap between intent (blocking via robots.txt) and outcome (AI bots still representing 24% of traffic) underlines the need for infrastructure that gives publishers real control, not just declarations.

What this means for publishers

The data suggests three things. First, blocking via robots.txt alone is insufficient for managing AI access to content. Second, the industry needs better tooling that goes beyond binary allow/deny decisions. Third, the scale of AI bot traffic is significant enough that ignoring it has real implications for server costs, analytics accuracy, and content distribution strategy.

Content Bridge provides publishers with granular bot management at the webserver, CDN, and log level, giving editorial and product teams the visibility and control they need.

The full report with all charts is available as a downloadable carousel on our LinkedIn page.

The AI Bots Insights Report is published monthly by Content Bridge, based on real traffic data from our DACH publisher network. For questions or to contribute your data, reach out at nadeemqureshi@content-bridge.eu.