Industry Pulse is live, and here's what August looked like

Most publishers can see which bots are hitting their site. What they can't see is whether those numbers are normal.

Is a 20% AI bot share high? Is everyone else blocking that crawler, or just you? Until now the honest answer was that nobody really knew, because there was nothing to compare against.

What Industry Pulse does

Industry Pulse is now live in the Content Bridge platform. It puts your own bot traffic next to benchmark data from the wider market and aggregated figures from platform customers.

So instead of staring at your own numbers in isolation, you can see where you sit. Which crawlers are hitting you harder than average. Which ones most publishers have already shut out. Whether your AI bot share is unusual or completely ordinary for a site your size.

It's available to platform customers now.

A few things from the August report

Alongside the platform feature, our monthly AI Bots Insights Report is out. Some of what stood out.

robots.txt is everywhere, llms.txt is growing. 98.6% of the top 500 German publishers have a robots.txt file. 11.6% have adopted llms.txt. First publishers start to host RSL licensing files.

The most blocked bots are all AI crawlers. CCBot, GPTBot, Bytespider and ClaudeBot lead the list of bots blocked via robots.txt. Publishers are using the one lever they have, even though robots.txt is a request rather than an enforcement mechanism. More and more crawlers bypass it.

Search still dominates. AI-related bots account for roughly 20% of bot traffic, but classic Search and Indexing is still the single biggest category at 28.3%. AI crawling is significant and growing, but it hasn't taken over.

The AI Bots Insights Report is published monthly by Content Bridge. Questions or want to contribute your data? Get in touch at nadeemqureshi@content-bridge.eu.