State of AI crawler blocking 2026: we checked 123 sites' robots.txt
We ran 123 well-known sites' robots.txt through our own matcher. The headline isn't a single number — it's the split: news publishers block AI crawlers at 90%, developer sites at 0%.