3 ms·
> What gives you the right to load their content on terms other than theirs? robots.txt for one thing.
by alex_young 8y ago
> What gives you the right to load their content on terms other than theirs?
robots.txt for one thing.
- TeMPOraL 8y agorobots.txt is a gentlemen's agreement between a site and search engine running automated crawlers. It has no meaning for regular visitors.