3 ms·
Did anyone look at the robots.txt? It looks to me like it doesn't block anything - it does have some specific settings tailored for various big-boy search engi
by dedward 16y ago
Did anyone look at the robots.txt?
It looks to me like it doesn't block anything - it does have some specific settings tailored for various big-boy search engines, with a catch-all rule at the end for those not otherwise defined.
It also says right there at the top "# If you operate a search engine and would like to crawl Quora, please
# email info@quora.com. Thanks.
"
So they're trying to get the right data to the right search engines the right way - makes sense to me.
User-agent: *
Allow: /$
Allow: /about$
Allow: /about/
Allow: /jobs$
Allow: /challenges$
Allow: /press$
Allow: /login$
Allow: /login/
Allow: /signup$
- jarek 16y agoThe important part is Disallow: / right after the allows.