4 ms·
> robots.txt is a service to crawlers to help them avoid wasting time I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping
by shermantanktop 20d ago
> robots.txt is a service to crawlers to help them avoid wasting time
I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping crawlers be efficient.
- Zambyte 20d agoWell that's what they're doing.