Speaking from experience, this is why robots.txt files don't work; they've become anti-competitive in nature. If you actually build code to obey them you get l

Speaking from experience, this is why robots.txt files don't work; they've become anti-competitive in nature. If you actually build code to obey them you get locked out of actual sites, while the real scrapers you don't want don't listen to robots.txt anyways. If a site really doesn't want you there, they can block you at the IP/User-Agent level, which is what Quora will end up doing.