> I don't think one can make the general statement that "when you publish a website, you intend for it to be aggregated." People have attempted to sue Google fo

> I don't think one can make the general statement that "when you publish a website, you intend for it to be aggregated." People have attempted to sue Google for exactly that sort of aggregation. And there are plenty of sites that disallow crawling via their robots.txt (for obvious reasons, we don't see them as often as they aren't reachable via search engines). I know, I run such website myself (it’s where I upload family photos to share with other family members. Basically the good bits of Facebook but without having to deal with Facebook). However your point is that my generalisation was unsound and thus far you’ve just listed edge cases. Edge cases I acknowledge, sure. But edge cases are allowed to exist when one uses generalisations because the purpose of a generalisation is saying “in general” (ie most of the time) and not “all of the time”. So while your points are valid, they’re not a rebuttal. > And certainly in the era when search engines were being developed (in the early '90s, when robots.txt came into existence), there was no such aggregation expectation because the technology wasn't there to do such aggregation yet Aggregated index sites still existed back then. The difference between then and now was that they were curated by humans. My first website was published in 1994 so I do remember the early web pretty well :) > On the flip side, Google operates under the general assumption that in the absence of reasons to the contrary, there is nothing wrong with info aggregation. "Aggregation is okay" is their default stance. Organizing the world's information and making it universally accessible are tent-poles in their mission statement. I’m well aware of Google’s original mission. I don’t think it’s fair to say this remains their current mission (maybe one of their missions but they’re a much larger organisation now and each department will have their own goals and measurements). However the point of this discussion wasn’t what Google’s past nor present mission was/is, it’s whether it’s ethically sound in the very specific context of SSID aggregation.