5 ms·

Did anyone actually go through the robots.txt file? http://www.quora.com/robots.txt http://www.quora.com/robots.txt The first two lines are: # If you oper

by niyazpk 16y ago

Did anyone actually go through the robots.txt file? http://www.quora.com/robots.txt http://www.quora.com/robots.txt The first two lines are: # If you operate a search engine and would like to crawl Quora, please # email info@quora.com. Thanks. Looks sensible to me.

joe_the_user 16y ago
Ah, but do you know what the results of that emailing have been? Five bots are listed. Perhaps they're the only one who bothered to email. Or perhaps not.
- dedward 16y ago
  What about the rule that matches "*" at the end? That's there for all the rest - the others are just fine-tuned for mutual benefit. There is no rule in there that denys other search engines, just some rules that set some slightly different rules for different ones. EDIT: I see now -therule at the end is rather specific, and wouldn't allow crawling the entire site. Still - without actual complaints from someone who contacted them and got no response, I see no evil.
  - jdp23 16y ago
    Did you actually read the article? EDIT: my bad. I got confused between Facebook and Quora
  - joe_the_user 16y ago
    Could you clarify what the line: Disallow: / means?
    - michaelbuckbee 16y ago
      It's indicating the root of the site. Documentation for further reading at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
      - joe_the_user 16y ago
        Indeed, It seemed self-explanatory here. I was being perhaps a bit sarcastic...
JoachimSchipper 16y ago
Yeah, because mailing every two-cent site to get permission to crawl is a sensible way to build a search engine. Quora should just get over themselves.
- pclark 16y ago
  Ironically I believe that this is to protect the users privacy on the Quora site. They have an option to hide your username from your answers, so if someone were to Google your name your Quora answers wouldn't appear.
  - jcr 16y ago
    If one wants to be an idiot or an asshole on the Internet, then one is free to do so. If one wants to attach their real name, address or some other identifiable bit of information to such outbursts, then once again, one is free to do so. The same goes for life off of the Internet, which of course could be recorded or reported, and subsequently placed on the Internet. The real problem is people mistakenly believe they can secretly operate with impunity in an age of increasing transparency.
    - pclark 16y ago
      You've successfully identified a fundamental problem with society, and it is by no means limited to Quora. I think the search engine privacy feature is a fantastic innovation from Quora to help absolve potential problems in the future. (eg: you google my name and see my Quora answers on Sex)
      - jcr 16y ago
        The supposed "potential problems" are fictitious. If you wrote public answers on questions of sex, why should they be hidden? --You made your answers public intentionally and also provided the answers intentionally so others would hopefully learn from them. Additionally, you did both while knowing your name is attached to them. In other words, you want to be known both for and by your contributions. If you did not wish to be known for and by your public contributions, then the only answer is to either not make public contributions, or go to rather extreme lengths to only make public contributions anonymously. The latter fraught with plenty of caveats to maintain anonymity correctly, so the only real answer is to only make public contributions you would want attributed to you. Back in the 80's when Chico State was the number 1 party school in the US, recruiters would show up in droves looking to get new employees. The unstated reason was very simple; if someone could graduate from Chico, it pretty much proved the person could not only get things done but also have a really fun time doing it. Who would you rather work with? --A fiercely academic but socially limited graduate, or the guy who can get stuff done and also do keg stands? If I searched on your name and found your public opinions, I'd think far more of you for having the stones to make your own contributions than I would be concerned about your predilection for midget porn. Of course, the trouble with trying to be non-judgmental and unbiased is assuming others will do the same.

by niyazpk 16y ago

joe_the_user 16y ago

Ah, but do you know what the results of that emailing have been? Five bots are listed. Perhaps they're the only one who bothered to email. Or perhaps not.

dedward 16y ago
What about the rule that matches "*" at the end? That's there for all the rest - the others are just fine-tuned for mutual benefit. There is no rule in there that denys other search engines, just some rules that set some slightly different rules for different ones. EDIT: I see now -therule at the end is rather specific, and wouldn't allow crawling the entire site. Still - without actual complaints from someone who contacted them and got no response, I see no evil.
- jdp23 16y ago
  Did you actually read the article? EDIT: my bad. I got confused between Facebook and Quora
- joe_the_user 16y ago
  Could you clarify what the line: Disallow: / means?
  - michaelbuckbee 16y ago
    It's indicating the root of the site. Documentation for further reading at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
    - joe_the_user 16y ago
      Indeed, It seemed self-explanatory here. I was being perhaps a bit sarcastic...

JoachimSchipper 16y ago

Yeah, because mailing every two-cent site to get permission to crawl is a sensible way to build a search engine. Quora should just get over themselves.

pclark 16y ago
Ironically I believe that this is to protect the users privacy on the Quora site. They have an option to hide your username from your answers, so if someone were to Google your name your Quora answers wouldn't appear.
- jcr 16y ago
  If one wants to be an idiot or an asshole on the Internet, then one is free to do so. If one wants to attach their real name, address or some other identifiable bit of information to such outbursts, then once again, one is free to do so. The same goes for life off of the Internet, which of course could be recorded or reported, and subsequently placed on the Internet. The real problem is people mistakenly believe they can secretly operate with impunity in an age of increasing transparency.
  - pclark 16y ago
    You've successfully identified a fundamental problem with society, and it is by no means limited to Quora. I think the search engine privacy feature is a fantastic innovation from Quora to help absolve potential problems in the future. (eg: you google my name and see my Quora answers on Sex)
    - jcr 16y ago
      The supposed "potential problems" are fictitious. If you wrote public answers on questions of sex, why should they be hidden? --You made your answers public intentionally and also provided the answers intentionally so others would hopefully learn from them. Additionally, you did both while knowing your name is attached to them. In other words, you want to be known both for and by your contributions. If you did not wish to be known for and by your public contributions, then the only answer is to either not make public contributions, or go to rather extreme lengths to only make public contributions anonymously. The latter fraught with plenty of caveats to maintain anonymity correctly, so the only real answer is to only make public contributions you would want attributed to you. Back in the 80's when Chico State was the number 1 party school in the US, recruiters would show up in droves looking to get new employees. The unstated reason was very simple; if someone could graduate from Chico, it pretty much proved the person could not only get things done but also have a really fun time doing it. Who would you rather work with? --A fiercely academic but socially limited graduate, or the guy who can get stuff done and also do keg stands? If I searched on your name and found your public opinions, I'd think far more of you for having the stones to make your own contributions than I would be concerned about your predilection for midget porn. Of course, the trouble with trying to be non-judgmental and unbiased is assuming others will do the same.