3 ms·

There’s a big difference too in that when you publish a website, you intend for it to be aggregated. Whereas people don’t intend their SSID to be aggregated. I

There’s a big difference too in that when you publish a website, you intend for it to be aggregated. Whereas people don’t intend their SSID to be aggregated. It’s more similar to walking along the street compiling a list of what colour curtains people put up in their window. Sure you’ve technically got it on display but people don’t really expect that information to be aggregated and shared. Ease dropping on a conversation on the bus or a train is another example. You can’t really complain if someone overhears but that’s a far cry from someone then publishing what they overhear

jjulius 5y ago
>There's a big difference too in that when you publish a website, you intend for it to be aggregated. You're framing this as an absolute and that's just not true. Were it the case, robots.txt wouldn't exist and I wouldn't be using it.
- laumars 5y ago
  I’m not framing it as an absolute. I’m generalising. I accept edge cases exist but my point stands for the majority use cases. Also you shouldn’t really be using robots.txt any more unless it’s a simple “Disallow: /“ because ironically bad actors use it to decide what URIs to hit. If you have content up you want to limit access to, you’re much better off putting it behind an auth layer (even if it’s just simple HTTP auth + fail2ban)
shadowgovt 5y ago
I don't think one can make the general statement that "when you publish a website, you intend for it to be aggregated." People have attempted to sue Google for exactly that sort of aggregation. And there are plenty of sites that disallow crawling via their robots.txt (for obvious reasons, we don't see them as often as they aren't reachable via search engines). And certainly in the era when search engines were being developed (in the early '90s, when robots.txt came into existence), there was no such aggregation expectation because the technology wasn't there to do such aggregation yet. Linking from one document to another has always been a part of the standard, but I don't think one could assume anything one published would end up in a vast auto-generated index. On the flip side, Google operates under the general assumption that in the absence of reasons to the contrary, there is nothing wrong with info aggregation. "Aggregation is okay" is their default stance. Organizing the world's information and making it universally accessible are tent-poles in their mission statement.
- laumars 5y ago
  > I don't think one can make the general statement that "when you publish a website, you intend for it to be aggregated." People have attempted to sue Google for exactly that sort of aggregation. And there are plenty of sites that disallow crawling via their robots.txt (for obvious reasons, we don't see them as often as they aren't reachable via search engines). I know, I run such website myself (it’s where I upload family photos to share with other family members. Basically the good bits of Facebook but without having to deal with Facebook). However your point is that my generalisation was unsound and thus far you’ve just listed edge cases. Edge cases I acknowledge, sure. But edge cases are allowed to exist when one uses generalisations because the purpose of a generalisation is saying “in general” (ie most of the time) and not “all of the time”. So while your points are valid, they’re not a rebuttal. > And certainly in the era when search engines were being developed (in the early '90s, when robots.txt came into existence), there was no such aggregation expectation because the technology wasn't there to do such aggregation yet Aggregated index sites still existed back then. The difference between then and now was that they were curated by humans. My first website was published in 1994 so I do remember the early web pretty well :) > On the flip side, Google operates under the general assumption that in the absence of reasons to the contrary, there is nothing wrong with info aggregation. "Aggregation is okay" is their default stance. Organizing the world's information and making it universally accessible are tent-poles in their mission statement. I’m well aware of Google’s original mission. I don’t think it’s fair to say this remains their current mission (maybe one of their missions but they’re a much larger organisation now and each department will have their own goals and measurements). However the point of this discussion wasn’t what Google’s past nor present mission was/is, it’s whether it’s ethically sound in the very specific context of SSID aggregation.

jjulius 5y ago

>There's a big difference too in that when you publish a website, you intend for it to be aggregated. You're framing this as an absolute and that's just not true. Were it the case, robots.txt wouldn't exist and I wouldn't be using it.

laumars 5y ago
I’m not framing it as an absolute. I’m generalising. I accept edge cases exist but my point stands for the majority use cases. Also you shouldn’t really be using robots.txt any more unless it’s a simple “Disallow: /“ because ironically bad actors use it to decide what URIs to hit. If you have content up you want to limit access to, you’re much better off putting it behind an auth layer (even if it’s just simple HTTP auth + fail2ban)

shadowgovt 5y ago

I don't think one can make the general statement that "when you publish a website, you intend for it to be aggregated." People have attempted to sue Google for exactly that sort of aggregation. And there are plenty of sites that disallow crawling via their robots.txt (for obvious reasons, we don't see them as often as they aren't reachable via search engines). And certainly in the era when search engines were being developed (in the early '90s, when robots.txt came into existence), there was no such aggregation expectation because the technology wasn't there to do such aggregation yet. Linking from one document to another has always been a part of the standard, but I don't think one could assume anything one published would end up in a vast auto-generated index. On the flip side, Google operates under the general assumption that in the absence of reasons to the contrary, there is nothing wrong with info aggregation. "Aggregation is okay" is their default stance. Organizing the world's information and making it universally accessible are tent-poles in their mission statement.

laumars 5y ago
> I don't think one can make the general statement that "when you publish a website, you intend for it to be aggregated." People have attempted to sue Google for exactly that sort of aggregation. And there are plenty of sites that disallow crawling via their robots.txt (for obvious reasons, we don't see them as often as they aren't reachable via search engines). I know, I run such website myself (it’s where I upload family photos to share with other family members. Basically the good bits of Facebook but without having to deal with Facebook). However your point is that my generalisation was unsound and thus far you’ve just listed edge cases. Edge cases I acknowledge, sure. But edge cases are allowed to exist when one uses generalisations because the purpose of a generalisation is saying “in general” (ie most of the time) and not “all of the time”. So while your points are valid, they’re not a rebuttal. > And certainly in the era when search engines were being developed (in the early '90s, when robots.txt came into existence), there was no such aggregation expectation because the technology wasn't there to do such aggregation yet Aggregated index sites still existed back then. The difference between then and now was that they were curated by humans. My first website was published in 1994 so I do remember the early web pretty well :) > On the flip side, Google operates under the general assumption that in the absence of reasons to the contrary, there is nothing wrong with info aggregation. "Aggregation is okay" is their default stance. Organizing the world's information and making it universally accessible are tent-poles in their mission statement. I’m well aware of Google’s original mission. I don’t think it’s fair to say this remains their current mission (maybe one of their missions but they’re a much larger organisation now and each department will have their own goals and measurements). However the point of this discussion wasn’t what Google’s past nor present mission was/is, it’s whether it’s ethically sound in the very specific context of SSID aggregation.