Search engines can be a real PITA for a site. They can take a site down. When it's my site - I don't have to let you, or anyone else, index it. We can do it

Search engines can be a real PITA for a site. They can take a site down. When it's my site - I don't have to let you, or anyone else, index it. We can do it nicely through robots.txt, or cold-war style through firewalls and script and tarpits and who knows what else. So - assuming Quora took the time to put what they did in their robots.txt, in a world where many sites still don't bother at all - one can assume they are paying close attention to the business value they are extracting from search engine driven traffic.