4 ms·
What about the rule that matches "*" at the end? That's there for all the rest - the others are just fine-tuned for mutual benefit. There is no rule in there
by dedward 16y ago
What about the rule that matches "*" at the end? That's there for all the rest - the others are just fine-tuned for mutual benefit.
There is no rule in there that denys other search engines, just some rules that set some slightly different rules for different ones.
EDIT: I see now -therule at the end is rather specific, and wouldn't allow crawling the entire site. Still - without actual complaints from someone who contacted them and got no response, I see no evil.
- jdp23 16y agoDid you actually read the article? EDIT: my bad. I got confused between Facebook and Quora
- joe_the_user 16y agoCould you clarify what the line: Disallow: / means?
- michaelbuckbee 16y agoIt's indicating the root of the site. Documentation for further reading at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
- joe_the_user 16y agoIndeed, It seemed self-explanatory here. I was being perhaps a bit sarcastic...