36 ms·
That robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....
by napolux 14y ago
That robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....
- fein 14y agoI would hope that google, of all companies, would be capable of writing a crawler that ignores robots.txt comments. Other than that the guy only has one rule.
- zalew 14y agorobots this size aren't that unusual http://www.bbc.co.uk/robots.txt http://www.bbc.co.uk/robots.txt
- chayesfss 14y agowow, just realized this is a great way to find information about websites
- deleted 14y ago[deleted]
- thenextcorner 14y agoI think you just proved the point here: SEO can be very valuable. Any good SEO would point out to the BBC that having similar rules for User-agent: Googlebot-Mobile and User-agent: * is not very efficient.
- signifiers 14y agoI bet they are fairly liberal with their size limits: http://www.amazon.com/robots.txt http://www.amazon.com/robots.txt http://aws.amazon.com/robots.txt http://aws.amazon.com/robots.txt http://www.google.com/robots.txt http://www.google.com/robots.txt http://store.apple.com/robots.txt http://store.apple.com/robots.txt
- martin-adams 14y agoApple should use git instead of subversion :)
- SageRaven 14y agoAnybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?
- kanzure 14y agoThe robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.
- dsl 14y ago500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.