Anthropic clarifies how Claude bots crawl sites and how to block them Anthropic explains how its bots handle AI training, live queries, and search results, and what opting out means for visibility.

#Business #Reports
Anthropic details how Claude crawls sites · How to block the three separate user agents ilo.im/16ax7y by Danny Goodwin

_____
#AI #Claude #Crawlers #UserAgents #RobotsTxt #Content #Website #WebDev #Frontend #Backend

Waterfox - Open source web browser The web browser that respects your privacy

Of the bigger browsers, I think Safari is probably the closest to that right now, for all its flaws. It may be that Waterfox or some other Firefox fork with the AI garbage ripped out of it is better, I haven't delved into that.

https://www.waterfox.com

#Browsers #userAgents

AI Bots and Robots.txt There’s been a lot of discussion lately around AI crawlers and bots, which are used to train LLMs and/or fetch content on behalf of their users. In the past few weeks I’ve seen blog posts about the...

#Development #Findings
AI bots and robots.txt · How websites use robots.txt to set AI crawling rules ilo.im/166la4 by Paul Calvano

_____
#AI #Bots #Content #Website #UserAgents #RobotsTxt #Business #SEO #WebDev #Backend

So I made myself a #nginx config page containing lots of denied requests and sending them to a gzip forkbomb, and some little stuff to discourage script kiddies :
gist.github.com/garfieldairl...

Suggestions welcome !

For bad #useragents and IP addresses I suggest
github.com/mitchellkrog...