3 ms·
Isn't there a robots.txt equivalent for AI?
by sgt 2y ago
Isn't there a robots.txt equivalent for AI?
- simianparrot 2y agoIt’s not respected, there’s a lot of posts on HN about this. Just one quick reference: https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...
- Hamuko 2y agoYou'd probably have a better time using something like Cloudflare to block known scrapers than trust companies that pirate terabytes of books to train their models to obey a "please don't" text file. https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...