3 ms·
It's just straight up evil to run an indexing service that ignores robots.txt and to run more than one of these things that when they don't talk to each other.
by docgnome 17y ago
It's just straight up evil to run an indexing service that ignores robots.txt and to run more than one of these things that when they don't talk to each other. Even if the latter was stupidity, the former is just plain evil. It's not like they have an excuse of ignorance. This isn't some guy who just learned to program hacking something together. These are supposed to be "professional" programmers.
- chriseppstein 17y agoM$ never promised to not be evil.
- bockris 17y agoAnd it's nothing new. A couple of years ago I used to have a domain that I put a blanket disallow on. Live search indexed it but neither Yahoo or Google did.