2 ms·
It’s not respected, there’s a lot of posts on HN about this. Just one quick reference: https://www.tomshardware.com/tech-industry/artificial-intelligence/sever
by simianparrot 2y ago
It’s not respected, there’s a lot of posts on HN about this.
Just one quick reference: https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...