A daily robots.txt measurement across Germany’s top 1000 websites: which AI bots get locked out, how often — and what the 20 biggest sites do. All numbers are free to cite.
As of 3 July 2026 · Panel: Deutschland (CrUX Top-1000), 891 sites reachable
Share of reachable panel sites that effectively lock out each bot (robots.txt incl. server-side blocks where measured):
Every day the robots.txt of Germany’s top 1000 websites (CrUX panel) is measured; for 891 reachable sites the rules for 12 known AI crawlers are evaluated. A subset is additionally probed for server-side blocks (e.g. 403 for bot user agents).
“Effectively blocked” means: an explicit robots.txt rule against the bot or a verified server-side block. Sites without a rule count as not blocking — and robots.txt is a request, not a technical barrier.
Tracked since 3 July 2026. The page is recomputed from the current measurement with every site build.