2 ms·
> They copied data from every website in existence to populate their search engine despite copyright laws, encouraged some sort of robots.txt opt-out mechanism
by gumby 6y ago
> They copied data from every website in existence to populate their search engine despite copyright laws, encouraged some sort of robots.txt opt-out mechanism while also ignoring the (AI-hard) need to parse legal terms of use on each website which may not have allowed their access.
As did every other search engine, and a good thing too. In general the principle of “if you don’t want people to see something don’t post it publicly” is a good principle that predates computers.
As for robots.txt: don’t forget that it predated Google, even when g was just a grad school project. Nothing weird about it: we also specify how the http protocol works. Humans use “AI-hard” adaptive protocols and there’s no reason computers won’t in future, but they can’t yet and it’s not unreasonable to have to adapt to that.
There are many many reasons to criticize Google but I think your argument is not one of them.