3 ms·
But robots.txt provides minimums: don't scrape this page, don't refresh more than once every x, these crawlers are allowed this access, etc.
by thatthatis 13y ago
But robots.txt provides minimums: don't scrape this page, don't refresh more than once every x, these crawlers are allowed this access, etc.