3 ms·
This is doing the rounds on the net too: http://www.thegatesnotes.com/robots.txt http://www.thegatesnotes.com/robots.txt It doesn't mention Bing. Amusing, and
by pierrefar 17y ago
This is doing the rounds on the net too:
http://www.thegatesnotes.com/robots.txt http://www.thegatesnotes.com/robots.txt
It doesn't mention Bing. Amusing, and useless.
- walkon 17y agoWhy is it useless (I've never worked with robots.txt)? Are the /css and /js files not large enough to bother skipping?
- pierrefar 17y agoUseless in the sense it doesn't really need to mention Bing. Blocking /css and /js makes sense because they don't add value in being crawled. Not necessary, but doesn't hurt. Best practice: Always have a robots.txt file, even if it's empty.
- beza1e1 17y agoWhat is the reason for your Best Practice advice?
- mshafrir 17y agoLook at the comments... # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources.
- thwarted 17y agoIt's a cut and paste from some sample file, so this doesn't indicate anything much. google search for the text in the comments ( http://tinyurl.com/ygtuanh http://tinyurl.com/ygtuanh ) has 49 hits, there would be more if I had not joined the lines together, I'm sure.
- lolcraft 17y agoBing ignores robots.txt anyway: http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must-die.html http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must...