4 ms·Google Formalizes Robots Exclusion Protocol in Effort to Make It a Standard1 points by nfrankel 7y ago