4 ms·
No technical reason. Following robots.txt is purely self-enforced, crawlers don't even have to read it if they don't want to. It's just part of being a good net
by Volundr 11y ago
No technical reason. Following robots.txt is purely self-enforced, crawlers don't even have to read it if they don't want to. It's just part of being a good netizen.
- podgib 11y agoThanks. That's what I suspected - I'm just surprised that yandex is behaving like a good netizen :)