4 ms·
Doesn't excluding them in robots.txt solve the search engine problem?
by ikeboy 11y ago
Doesn't excluding them in robots.txt solve the search engine problem?
- _Understated_ 11y agoFollowing the rules laid out in a robots.txt file is optional. The reputable search engines tend to play by the rules but dodgy ones? Not so sure.
- ikeboy 11y agoThe dodgy ones don't have access to the URLs. The example was Skype links: if Microsoft scrapes those, they'll follow the rules and not make them available in searches. If you give your link to a dodgy search engine, you've lost.