2 ms·
> robots.txt is a service to crawlers to help them avoid wasting time I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping
by shermantanktop 2mo ago
> robots.txt is a service to crawlers to help them avoid wasting time
I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping crawlers be efficient.
- Zambyte 2mo agoWell that's what they're doing.