4 ms·
Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clea
by _zagj 1y ago
Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.
- chneu 1y agoNobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info. Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day. I think debian also recently spoke up about it.
- EPendragon 1y agoOne way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.