3 ms·
Sure, but Google's crawler (mostly) obeys robots.txt and other directives. It seems to be one of the most responsible crawlers this way. Here, Invidious is expl
by jcrawfordor 3y ago
Sure, but Google's crawler (mostly) obeys robots.txt and other directives. It seems to be one of the most responsible crawlers this way. Here, Invidious is explicitly being told to stop and refusing to do so.
In a lot of these situations there's not much wrong with doing it until you're told to stop. Invidious has just been told to stop.
Google used to have services that felt a lot more like this, like Google Images before it got aggressive about using cached versions and old, old Google Videos. But they got rid of all of those, I suspect because of thinking along these lines. It was very much an open legal issue around newspaper publishers, for example, and while Google is far from innocent they did make changes to mitigate their legal risk. Google seems to be pretty consistent in the idea that serving up content from someone else's service, in their products, is not acceptable. All of AMP kind of got built around that issue. And I don't think Google wanted it that way, but the legal landscape is the legal landscape. If Google is bound by these rules others have to be as well.