4 ms·
Following the rules laid out in a robots.txt file is optional. The reputable search engines tend to play by the rules but dodgy ones? Not so sure.
by _Understated_ 11y ago
Following the rules laid out in a robots.txt file is optional. The reputable search engines tend to play by the rules but dodgy ones? Not so sure.
- ikeboy 11y agoThe dodgy ones don't have access to the URLs. The example was Skype links: if Microsoft scrapes those, they'll follow the rules and not make them available in searches. If you give your link to a dodgy search engine, you've lost.