4 ms·
> I should be able to allow or disallow any type of web crawler/scraper i want. You're certainly allowed to try, but I don't see why indexers should be mandate
by ric2b 3y ago
> I should be able to allow or disallow any type of web crawler/scraper i want.
You're certainly allowed to try, but I don't see why indexers should be mandated to collaborate with you. They serve their users, not you.
- tympious 3y agoWill their users appreciate that they disregard the intent of the authors of what they index? I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.