You can't just scrape websites and ignore robots.txt, you will get into infinite loops. That doesn't mean people don't do it anyway, but "first pass" scraping i

You can't just scrape websites and ignore robots.txt, you will get into infinite loops. That doesn't mean people don't do it anyway, but "first pass" scraping it is more efficient to respect robots.txt, and then you can come in with a human to figure out if anything behind robots.txt is surreptitiously possible to scrape. On the other hand, Google etc. do not care enough, they just respect robots.txt because they don't need to index your site if you don't want them to, it hurts you more than it hurts them. DNT has no mutual benefit, the benefit is only to the end-user asking not to be tracked. The benefit is also not immediate, it's somewhere down the road. Ignoring robots.txt can immediately cause problems for both the client and server.