4 ms·

I was thinking about this line a lot. Because yes, "not all bots" and "not in all cases". But what gives them the right to do whatever with the sites if they c

I was thinking about this line a lot. Because yes, "not all bots" and "not in all cases". But what gives them the right to do whatever with the sites if they claim it's for security purposes? They are not law enforcement. Moreover, other bots are missing from the docs which they clearly state on the same page. How many other bots of theirs are ignoring robots.txt? I think we should hold Google to a higher standard than some random blogger (me).

arcfour 13d ago
...Because law enforcement does not have the interest, reach, or time to investigate every single cybercrime, and if they do it won't be timely - it will be after the fact, punishing the criminals, and not preventing harm. Google is in a unique situation to protect their users, and they also want to avoid serving search results that are malicious (yes, I know about their ads problem) - so they are proactive about scanning for sites that are malicious so they can avoid sending users to them.
- ArcHound 10d ago
  See, here's where I disagree. Plenty of people are in an unique position to help the law enforcement. And yet it's usually criminal to do so. Also, I don't want Google to be the police of the internet, especially when they felt the need to drop the "don't be evil" guideline.