3 ms·
And Yandex! Side note, I have friends that crawled a massive amount of the internet over several months for their own purposes.. at this point it's probably im
by talldatethrow 3y ago
And Yandex!
Side note, I have friends that crawled a massive amount of the internet over several months for their own purposes.. at this point it's probably impossible to exclude your site since tons of other people probably link to your site if it's at all of value.
- JohnFen 3y agoYeah, this is why robots.txt is garbage. Too many crawlers completely ignore it, so I stopped bothering with it (I have it set to block all bots, but I don't expect it to be effective.) Instead, I'd just keep an eye on my access logs and block obvious crawlers when I saw them.