3 ms·

Then what is the reason that `robots.txt` is so widely followed even without a legal requirement to do so?

• zinekeller 6y ago
  There is an incentive to not abuse their own server time serving robots, essentially creating the status quo. Meanwhile, advertisers value users' data like gold. Unlike server time, most advertisers are okay to collect their own data for advertising purposes, and this causes them to have an unwritten agreement to collect data. Unless there is a stronger stick to force them otherwise, it is in their best interests to collect data and you need to have active intervention to prevent it.
• Macha 6y ago
  Server operators are more capable of blocking badly behaving bots than users are of blocking adtech companies. Consequently you follow the limits in robots.txt or you risk getting blocked. On the other hand, if you follow the limits in DNT, you get nothing, and the consequences of not doing so are ... nothing, so companies did not follow that. (See also P3P for another previous attempt)
• lukeschlather 6y ago
  You can't just scrape websites and ignore robots.txt, you will get into infinite loops. That doesn't mean people don't do it anyway, but "first pass" scraping it is more efficient to respect robots.txt, and then you can come in with a human to figure out if anything behind robots.txt is surreptitiously possible to scrape. On the other hand, Google etc. do not care enough, they just respect robots.txt because they don't need to index your site if you don't want them to, it hurts you more than it hurts them. DNT has no mutual benefit, the benefit is only to the end-user asking not to be tracked. The benefit is also not immediate, it's somewhere down the road. Ignoring robots.txt can immediately cause problems for both the client and server.