4 ms·
Oh wow, I didn't realise my robots.txt joke had spread! I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws
by jonty 13y ago
Oh wow, I didn't realise my robots.txt joke had spread!
I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws/all/ http://www.wired.com/business/2010/08/robot-laws/all/), and have just been made aware that it appears to have spread across the internet:
https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fignoring%2Fhuman%2Forders" https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fi...
Favourite sites I've found with it so far: php.net, princeton.edu, nest.com, songkick.com.