3 ms·
That’s cool, if any scrapers would still respect the robots.txt that is
by yapyap 2y ago
That’s cool, if any scrapers would still respect the robots.txt that is
- bityard 2y agoThink of robots.txt as less of a no trespassing sign and more of a, "You can visit but here are the rules to follow if you don't want to get shot" sign.
- iterance 2y agoIf you do not respect the sign I shall be very cross with you. Very cross indeed. Perhaps I shall have to glare at you, yes, very hard. I think I shall glare at you. Perhaps if you are truly irritating I shall be forced to remove you from the premises for a bit.
- blacksmith_tb 2y agoThere's a lot of talk of deregulation in the air, maybe we'll see Gibson-esque Black Ice, where rude crawlers provoke an automated DoS, a new Wild West.
- dartos 2y agoI was surprised any ever did, honestly
- marginalia_nu 2y agoThey may or may not, though respecting robots.txt is a nice way of not having your IP range end up on blacklists. With cloudflare in particular, that can be a bit of a pain. They're pretty nice to deal with if you're upfront about what you are doing and clearly identify your bot, as well as register it with their bot detection. There's a form floating around somewhere for that.
- andrethegiant 2y agoFWIW, that’s why I’m working on a platform[1] to help devs deploy polite crawlers and scrapers out of the box that respect robots.txt (and 429s, Retry-After response headers, etc). It also happens to be entirely built on Cloudflare. [1] https://crawlspace.dev https://crawlspace.dev