4 ms·
It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore). It's just for us oldies who prefer the ethical web
by bayindirh 2y ago
It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore).
It's just for us oldies who prefer the ethical web where people respected the RFCs they have collectively written.
- pimlottc 2y agoIt’s one thing to know they don’t honor it, it’s another to know they intentionally abuse it
- johnea 2y agoYes, it is, and now we know both of those things don't we. What will be the consequence? My risky prediction: Absolutely Nothing, for neither goggle nor any of the other miscreants running the bot nets. And just to take a poke at the title of the original El Reg article: Do you really think the problem is that they just haven't "learned" yet how to "play nice"? I'm sure the great oracle goggle, just needs someone to poperly splain it to them.