4 ms·
Of course they would think it's dumb because some of robots.txt rules are counter to their objective which is to save the internet in all its glory. They should
by lcw 10y ago
Of course they would think it's dumb because some of robots.txt rules are counter to their objective which is to save the internet in all its glory. They shouldn't use robots.txt I agree. At the same time robots.txt is not worthless.
SEO is where robots.txt shines right now. It's not that people are trying to hide something it's because we don't want it to conflict with the content we actually want to promote.
- Walf 10y agoBingo. Everything I remember reading about robots.txt strongly emphasised that it's not a way to hide content – in case one's reasoning skills are lacking – and that it's main use is to prevent irrelevant or infinite content showing up in any indices.