3 ms·
Ah, but do you know what the results of that emailing have been? Five bots are listed. Perhaps they're the only one who bothered to email. Or perhaps not.
by joe_the_user 16y ago
Ah, but do you know what the results of that emailing have been?
Five bots are listed. Perhaps they're the only one who bothered to email. Or perhaps not.
- dedward 16y agoWhat about the rule that matches "*" at the end? That's there for all the rest - the others are just fine-tuned for mutual benefit. There is no rule in there that denys other search engines, just some rules that set some slightly different rules for different ones. EDIT: I see now -therule at the end is rather specific, and wouldn't allow crawling the entire site. Still - without actual complaints from someone who contacted them and got no response, I see no evil.
- jdp23 16y agoDid you actually read the article? EDIT: my bad. I got confused between Facebook and Quora
- joe_the_user 16y agoCould you clarify what the line: Disallow: / means?
- michaelbuckbee 16y agoIt's indicating the root of the site. Documentation for further reading at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
- joe_the_user 16y agoIndeed, It seemed self-explanatory here. I was being perhaps a bit sarcastic...