3 ms·
Why is it useless (I've never worked with robots.txt)? Are the /css and /js files not large enough to bother skipping?
by walkon 17y ago
Why is it useless (I've never worked with robots.txt)? Are the /css and /js files not large enough to bother skipping?
- pierrefar 17y agoUseless in the sense it doesn't really need to mention Bing. Blocking /css and /js makes sense because they don't add value in being crawled. Not necessary, but doesn't hurt. Best practice: Always have a robots.txt file, even if it's empty.
- beza1e1 17y agoWhat is the reason for your Best Practice advice?
- mshafrir 17y agoLook at the comments... # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources.