7.4%
Blocks an AI crawler
120 / 1,625
4.9%
Blocks GPTBot
79 / 1,625
4.7%
Blocks CCBot
77 / 1,625
3.9%
Blocks Google-Extended
64 / 1,625
AI crawlers blocked in robots.txt
| Crawler | Operator | UK brands | United Kingdom | Global |
|---|---|---|---|---|
| GPTBot | OpenAI (training) | 4.9% | 47% | 9.6% |
| ChatGPT-User | OpenAI (user fetch) | 1.9% | 44% | 5.2% |
| OAI-SearchBot | OpenAI (search) | 1.8% | 44% | 4% |
| ClaudeBot | Anthropic | 4.7% | 46% | 8.6% |
| anthropic-ai | Anthropic (legacy) | 4.1% | 46% | 7.4% |
| Google-Extended | Google (Gemini) | 3.9% | 45% | 7.7% |
| CCBot | Common Crawl | 4.7% | 4.9% | 9.4% |
| PerplexityBot | Perplexity | 3.1% | 44% | 5.1% |
| Bytespider | ByteDance | 5.4% | 4.6% | 9.1% |
| Applebot-Extended | Apple | 4.5% | 3.9% | 7.5% |
| Amazonbot | Amazon | 3.8% | 46% | 7.9% |
| meta-externalagent | Meta | 4.6% | 46% | 7.7% |
| cohere-ai | Cohere | 3.7% | 45% | 7.1% |
| Diffbot | Diffbot | 3.4% | 45% | 7% |
Share of sites whose robots.txt blocks the bot from the homepage by name.
By market
| Metric | UK brands | United Kingdom | Global |
|---|---|---|---|
| Blocks an AI crawler | 7.4% | 6% | 10% |
| Blocks GPTBot | 4.9% | 47% | 9.6% |
| Blocks ClaudeBot | 4.7% | 46% | 8.6% |
| Blocks CCBot | 4.7% | 4.9% | 9.4% |
| Blocks Google-Extended | 3.9% | 45% | 7.7% |
| Blocks PerplexityBot | 3.1% | 44% | 5.1% |
| Content-Signal in robots.txt | 1.8% | 2.2% | 4.1% |
| llms.txt | 21% | 17% | 15% |
Search bots blocked
- 1 PetalBot 6.2% 75
- 2 Baiduspider 4.3% 52
- 3 YandexBot 3.6% 43
- 4 Applebot 2.1% 25
- 5 SeznamBot 1.9% 23
- 6 Sogou 1.7% 21
- 7 Naverbot 1.7% 20
- 8 Yeti 1.5% 18
- 9 Bingbot 0.9% 11
- 10 DuckDuckBot 0.8% 10
Share of 1,211 sites in the sample.
SEO bots blocked
- 1 AhrefsBot 5.2% 63
- 2 MJ12bot 4.7% 57
- 3 DotBot 3.1% 37
- 4 BLEXBot 3% 36
- 5 SemrushBot 2.6% 31
- 6 DataForSeoBot 2.1% 26
- 7 rogerbot 2% 24
- 8 Barkrowler 2% 24
- 9 serpstatbot 1.7% 20
- 10 SiteAuditBot 1.7% 20
Share of 1,211 sites in the sample.
Our crawler: 1,359 sites · UK brands · updated 23 Oct 2026 · Methodology
Want your brand’s website benchmarked?
We compare your site with your category on security, speed, email and SEO — and fix what lags behind.