I mean pick a source. The top hit on Google (in my experiment segment) for “Google web crawler AI” is: https://www.theverge.com/24067997/robots-txt-ai-text-fil

I mean pick a source. The top hit on Google (in my experiment segment) for “Google web crawler AI” is: https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders https://www.theverge.com/24067997/robots-txt-ai-text-file-we... But the next 20 hits are just more ways Google is pushing the envelope on creative interpretations of robots.txt Yesterday I asked Sonnet 3.5 how to compete with Anthropic and it was like, point out that no serious science can be done without the ability to verify it. These things are pretty reliable at pointing out why they are bad news.