4 ms·
Robots.txt is a hint to indexers that their index would be better off ignoring certain resources, not an authorization system
by none_to_remain 4y ago
Robots.txt is a hint to indexers that their index would be better off ignoring certain resources, not an authorization system
- erik_seaberg 4y agoThis. It was created as a warning of an infinite subgraph of generated hyperlinks, because crawlers would get permanently lost and DDoS sites.