3 ms·Google's guide to Robots Exclusion Protocol (including new expires tag)1 points by nickb 19y ago