7.6%
Blocks an AI crawler
123 / 1,623
5.2%
Blocks GPTBot
84 / 1,623
5.2%
Blocks CCBot
85 / 1,623
4.7%
Blocks Google-Extended
76 / 1,623
AI crawlers blocked in robots.txt
| Crawler | Operator | UK brands | United Kingdom | Global |
|---|---|---|---|---|
| GPTBot | OpenAI (training) | 5.2% | 53% | 11% |
| ChatGPT-User | OpenAI (user fetch) | 1.6% | 49% | 4.9% |
| OAI-SearchBot | OpenAI (search) | 1.5% | 49% | 3.6% |
| ClaudeBot | Anthropic | 5.2% | 53% | 11% |
| anthropic-ai | Anthropic (legacy) | 4.7% | 52% | 9.6% |
| Google-Extended | Google (Gemini) | 4.7% | 52% | 9.7% |
| CCBot | Common Crawl | 5.2% | 5.8% | 11% |
| PerplexityBot | Perplexity | 2.7% | 49% | 4.7% |
| Bytespider | ByteDance | 5.7% | 5.5% | 11% |
| Applebot-Extended | Apple | 4.8% | 4.9% | 9.6% |
| Amazonbot | Amazon | 4.7% | 52% | 9.9% |
| meta-externalagent | Meta | 4.9% | 52% | 9.8% |
| cohere-ai | Cohere | 4.3% | 52% | 9.3% |
| Diffbot | Diffbot | 4% | 52% | 9.2% |
Share of sites whose robots.txt blocks the bot from the homepage by name.
By market
| Metric | UK brands | United Kingdom | Global |
|---|---|---|---|
| Blocks an AI crawler | 7.6% | 6.5% | 12% |
| Blocks GPTBot | 5.2% | 53% | 11% |
| Blocks ClaudeBot | 5.2% | 53% | 11% |
| Blocks CCBot | 5.2% | 5.8% | 11% |
| Blocks Google-Extended | 4.7% | 52% | 9.7% |
| Blocks PerplexityBot | 2.7% | 49% | 4.7% |
| Content-Signal in robots.txt | 3.3% | 3.7% | 6.8% |
| llms.txt | 16% | 13% | 15% |
Search bots blocked
- 1PetalBot8.4%84
- 2Baiduspider6.3%63
- 3YandexBot3.4%34
- 4SeznamBot2.1%21
- 5Applebot2%20
- 6Yeti1.6%16
- 7Sogou1.6%16
- 8Naverbot1.5%15
- 9DuckDuckBot0.8%8
- 10Bingbot0.7%7
Share of 1,003 sites in the sample.
SEO bots blocked
- 1AhrefsBot5%50
- 2MJ12bot4.9%49
- 3BLEXBot3.1%31
- 4DotBot3%30
- 5DataForSeoBot2.3%23
- 6SemrushBot2%20
- 7rogerbot1.8%18
- 8Barkrowler1.8%18
- 9SiteAuditBot1.7%17
- 10serpstatbot1.5%15
Share of 1,003 sites in the sample.
Our crawler: 1,277 sites · UK brands · updated 13 Nov 2026 · Methodology
Want your brand’s website benchmarked?
We compare your site with your category on security, speed, email and SEO — and fix what lags behind.