Advertisement · 728 × 90

Posts by StackScope

A wanted poster-style infographic titled 'Most (Not) Wanted' showing the 12 most blocked web crawlers based on robots.txt analysis of 3,235 sites. Top 4 with large logos: Common Crawl (15%), Bytespider by ByteDance (15%), GPTBot by OpenAI (15%), and Google Extended (14%). Below in a lineup: Amazonbot (14%), ClaudeBot by Anthropic (13%), Applebot Extended (13%), Meta (12%), Cloudflare Renderer (12%), Anthropic AI (1%), Apache Nutch (1%), and Cohere (1%). A faded red 'BLOCKED' stamp is overlaid diagonally across the poster.

A wanted poster-style infographic titled 'Most (Not) Wanted' showing the 12 most blocked web crawlers based on robots.txt analysis of 3,235 sites. Top 4 with large logos: Common Crawl (15%), Bytespider by ByteDance (15%), GPTBot by OpenAI (15%), and Google Extended (14%). Below in a lineup: Amazonbot (14%), ClaudeBot by Anthropic (13%), Applebot Extended (13%), Meta (12%), Cloudflare Renderer (12%), Anthropic AI (1%), Apache Nutch (1%), and Cohere (1%). A faded red 'BLOCKED' stamp is overlaid diagonally across the poster.

We analysed the robots.txt of 3,000+ new product launches.
Here's who's most (not) wanted.

Common Crawl takes the crown. More insights at stackscope.dev/insights

#AI #WebCrawlers #RobotsTxt #BuildInPublic #IndieHackers

1 day ago 4 0 0 0