13 ms·

Google Data Collection research

145 points by pjf 8y ago

pjf 8y ago
The site seems slow/down now. A cached copy is available, obviously, on Google :) https://webcache.googleusercontent.com/search?q=cache:m-ItC3KnszwJ:https://digitalcontentnext.org/blog/2018/08/21/google-data-collection-research/ https://webcache.googleusercontent.com/search?q=cache:m-ItC3...
- deleted 8y ago
  [deleted]
- exikyut 8y ago
  I asked myself, why on earth do websites have to be slow when they're under heavy load? To be fair, the technical reason is that the modern web (circa last 3-5 years or so?) has mostly switched over to demand scaling, and this is why hug-of-death situations are comparatively rare now. If you have to be super-economical, and even setting an upper spending limit is uncomfortable, set your site up to ping you when/if it's falling apart so you can pull the trigger on ordering more cores yourself. Maybe put a super easy "order more cores" button in the ping, make 2:30AM disasters less terrible. Of course, technical reasons aside, my initial unimpressedness (or, I must admit, indignation - ha) was directed at whoever this group's ISP is. Clearly the ISP is not providing enough [demand] bandwidth. [Edit: After reading some comments below I think I should insert a preface/prefix here and say that it's entirely possible the website itself is misconfigured. I think I got sufficiently indignant that I didn't factor this in, and this was definitely an oversight. I don't know the situation, and it's unfair to squarely blame the ISP. Original message continues unedited...] So, FWIW, this group is using mediatemple. (Found by a simple google of the IP and careful untangling of the results.) ...Well that's a name I've not heard in years. And it made me wonder - do MT have an answer to demand scaling? I'm not able to discern either way from a quick browse of their website (including the "managed wordpress hosting" section). If they don't (and they seem to be stuck in the VPS days), ouch. If they do have demand scaling, then I think it's unprofessional to offer configurations that don't handle situations like this. In any case, a "504 Gateway timeout" with a tiny "nginx" underneath is going to make people go "hmm, who's this person's ISP" and knock a few points off whatever the answer to that question is. I wouldn't want that happening to me or anyone I build websites for. [Edit, in the same vein as the edit above: the conclusion above is plausible, but incomplete. Again, it's necessary to consider all angles.]
  - r3bl 8y ago
    I'm always wondering the same. As a pretty noob sysadmin, my dead cheap[0] second-hand dedicated server was able to handle being on the HN homepage for 11ish hours without breaking a sweat. No CDN was involved what so ever, all content was going directly to my single point of failure. Yet, I have to resort to looking at cashed versions of the submissions from companies that have way more resources than I could ever imagine to have. [0] Price of the cheapest Media Temple VPS offering.
    - Rjevski 8y ago
      Looking at this page's source it seems like Wordpress (or rather Shitpress) is involved, so this downtime is not surprising at all considering how many resources that pile of garbage uses. But yeah, a properly optimized site can handle being on the HN front-page on a single server with no issues.
      - SquareWheel 8y ago
        There's plenty of caching plugins that make Wordpress run just fine at higher traffic. This site likely just doesn't have one installed.
        Rjevski 8y ago
        Would you buy a new car if it needed duct tape just so it can stay in one piece while driving on the highway? So why would you think it's acceptable for a CMS to require third-party "cache" plugins just to be able to handle traffic?
        exikyut 8y ago
        Computers and cars are both solutions to problems, but the complexity of computers is yet to be routinely contained without targeted effort. This is especially true of the Web. Wordpress is the single most installed CMS (and kitchen sink, at this point), it's a massive attack target, and it's installed by people who don't really know what they're doing. Between being a kitchen sink (and having to deal with the internal/external API/architectural [compatibility] baggage that comes with any long-lived implementation) and having to be the CMS equivalent of Windows/macOS, it's a... difficult ask for a WP installation with (say) 30 poorly-constructed addons to to run smoothly. Possible, yes, absolutely, but it'll need tuning. Caching is simply the simplest bandaid that can be tacked on. If the caching is tuned properly, it'll fix everything else. Yes, this will mean the underlying configuration/state will be the digital equivalent of a giant wound you just want to clean out and fix up, but, err... it works. :/
arnjerobben 8y ago
PDF: https://digitalcontentnext.org/wp-content/uploads/2018/08/DCN-Google-Data-Collection-Paper.pdf https://digitalcontentnext.org/wp-content/uploads/2018/08/DC...
econ4all 8y ago
It's interesting how the media cartel front thinks it's safe to drop all pretense and just post their opposition research like that. http://digitalcontentnext.org/membership/members http://digitalcontentnext.org/membership/members http://digitalcontentnext.org/about/overview/ http://digitalcontentnext.org/about/overview/ At the very least it reflects poorly on their member's coverage of FB/google.
- TangoTrotFox 8y ago
  I don't think it should matter who simple falsifiable statements of facts come from. 'Adversarial forces' would be the ones most motivated to discover these bits of information, and they're incredibly valuable. I mean this is an experiment anybody with a couple of phones and a dash of technical ability could carry out themselves. I would contrast this against the sort of macro-level research that involves modeling and other more subjective components subject to extensive 'tuning.' For instance the 'Uber drivers earn $3/hour' piece that ran a while back. The extensive modeling and obscured assumptions make it much easier to create fake conclusions that are very difficult to falsify.
  - deleted 8y ago
    [deleted]
- zepto 8y ago
  Who cares? What matters is whether it is true or not. Are you just trying to smear the information with an ad hominem, or do you have reason to believe it is inaccurate?
  - econ4all 8y ago
    It matters because the manner in which any "findings" are presented has a big role in shaping public opinion and in this case the opinion shapers are purpofuly distorting finding to attack an advesary. See the WSJ's false report that google let's developers read user emails, it's technically true but that only happens after the user allows it via an explicit permission screen. Or the AP's claim that google keeps track of user location ever after location history is turned off, which again is technically true but that is because location history is just a location feature and not the master switch for all location features. In both cases other media outlets repeated the claims without further commentery of research and this cartel might be the reason.
close04 8y ago
People will look fondly to the times when they thought Microsoft's telemetry was the worst privacy issue to hit them...
- fartcannon 8y ago
  Microsoft has not stopped spying, even now. They will likely be emboldened by their competitors exposure. Slowly boiling us all.
  - close04 8y ago
    Nice way of missing the point. I don't expect MS will ever stop. There's no incentive, people already proved that there's no treatment so bad that they will put their foot down and demand a change. Google opened the way for collecting massive amounts of data by any means possible (including and not limited to tricking the user into thinking they aren't). And people seem pretty at ease with this, comparatively few are as angered by the revelations these days as they were about MS's relatively harmless telemetry. So going back, at some point you'll miss the times when you thought MS collecting data about your hardware and drivers was "the evilest evil".
    - fartcannon 8y ago
      It might just be me, but I believe most people thought Google was a worse violator prior to Windows 10. Some (probably naive) people used to think Windows was the last bastion of non-Linux hope.
      - close04 8y ago
        Not really. MS has (had) a very crappy image that they worked really hard to build during the early 2000s. That reputation stuck. And it doesn't help that most people are ignorant and are always willing to ignore an informed opinion over one that sounds cool. MS and Windows 10 are still far behind Google in the data collection game. Especially since they put that privacy wizard front and center at the first setup. But as I said, every second you spend defending Google and throwing shade at MS because it feels like the cool thing to do, Google adds another data collection item in their policy, hides yet another setting under 5 layers of options, and tricks you again into thinking that toggling a switch to off actually disables it. It's not like they did exactly that for 10 years now to the point where they are by far the largest private data collection effort in the history of man kind. And yes, I'm sure MS will follow suit. Why not? Google seems to be doing just fine.
throwawaymanbot 8y ago
Look how quick the comments thread on this research got Hijacked in to site performance..
gooftop 8y ago
In some ways not surprised, but its still a LOT of data (in sheer volume). Does Google even keep the data and analyze it, or is much of it thrown away as digital exhaust? Anyone know?
OnlyRepliesToBS 8y ago
looks class actiony
yuhong 8y ago
My favorite is trying to trace the problems back to Larry/Sergey, which is why I wrote the essay. Worth mentioning that one of the hardest parts of writing it was tracing the history of things like Google Analytics.
- yuhong 8y ago
  I forgot to mention that I had an Ask HN on this: https://news.ycombinator.com/item?id=17447280 https://news.ycombinator.com/item?id=17447280

145 points by pjf 8y ago

pjf 8y ago

The site seems slow/down now. A cached copy is available, obviously, on Google :) https://webcache.googleusercontent.com/search?q=cache:m-ItC3KnszwJ:https://digitalcontentnext.org/blog/2018/08/21/google-data-collection-research/ https://webcache.googleusercontent.com/search?q=cache:m-ItC3...

deleted 8y ago
[deleted]
exikyut 8y ago
I asked myself, why on earth do websites have to be slow when they're under heavy load? To be fair, the technical reason is that the modern web (circa last 3-5 years or so?) has mostly switched over to demand scaling, and this is why hug-of-death situations are comparatively rare now. If you have to be super-economical, and even setting an upper spending limit is uncomfortable, set your site up to ping you when/if it's falling apart so you can pull the trigger on ordering more cores yourself. Maybe put a super easy "order more cores" button in the ping, make 2:30AM disasters less terrible. Of course, technical reasons aside, my initial unimpressedness (or, I must admit, indignation - ha) was directed at whoever this group's ISP is. Clearly the ISP is not providing enough [demand] bandwidth. [Edit: After reading some comments below I think I should insert a preface/prefix here and say that it's entirely possible the website itself is misconfigured. I think I got sufficiently indignant that I didn't factor this in, and this was definitely an oversight. I don't know the situation, and it's unfair to squarely blame the ISP. Original message continues unedited...] So, FWIW, this group is using mediatemple. (Found by a simple google of the IP and careful untangling of the results.) ...Well that's a name I've not heard in years. And it made me wonder - do MT have an answer to demand scaling? I'm not able to discern either way from a quick browse of their website (including the "managed wordpress hosting" section). If they don't (and they seem to be stuck in the VPS days), ouch. If they do have demand scaling, then I think it's unprofessional to offer configurations that don't handle situations like this. In any case, a "504 Gateway timeout" with a tiny "nginx" underneath is going to make people go "hmm, who's this person's ISP" and knock a few points off whatever the answer to that question is. I wouldn't want that happening to me or anyone I build websites for. [Edit, in the same vein as the edit above: the conclusion above is plausible, but incomplete. Again, it's necessary to consider all angles.]
- r3bl 8y ago
  I'm always wondering the same. As a pretty noob sysadmin, my dead cheap[0] second-hand dedicated server was able to handle being on the HN homepage for 11ish hours without breaking a sweat. No CDN was involved what so ever, all content was going directly to my single point of failure. Yet, I have to resort to looking at cashed versions of the submissions from companies that have way more resources than I could ever imagine to have. [0] Price of the cheapest Media Temple VPS offering.
  - Rjevski 8y ago
    Looking at this page's source it seems like Wordpress (or rather Shitpress) is involved, so this downtime is not surprising at all considering how many resources that pile of garbage uses. But yeah, a properly optimized site can handle being on the HN front-page on a single server with no issues.
    - SquareWheel 8y ago
      There's plenty of caching plugins that make Wordpress run just fine at higher traffic. This site likely just doesn't have one installed.
      - Rjevski 8y ago
        Would you buy a new car if it needed duct tape just so it can stay in one piece while driving on the highway? So why would you think it's acceptable for a CMS to require third-party "cache" plugins just to be able to handle traffic?
        exikyut 8y ago
        Computers and cars are both solutions to problems, but the complexity of computers is yet to be routinely contained without targeted effort. This is especially true of the Web. Wordpress is the single most installed CMS (and kitchen sink, at this point), it's a massive attack target, and it's installed by people who don't really know what they're doing. Between being a kitchen sink (and having to deal with the internal/external API/architectural [compatibility] baggage that comes with any long-lived implementation) and having to be the CMS equivalent of Windows/macOS, it's a... difficult ask for a WP installation with (say) 30 poorly-constructed addons to to run smoothly. Possible, yes, absolutely, but it'll need tuning. Caching is simply the simplest bandaid that can be tacked on. If the caching is tuned properly, it'll fix everything else. Yes, this will mean the underlying configuration/state will be the digital equivalent of a giant wound you just want to clean out and fix up, but, err... it works. :/

arnjerobben 8y ago

PDF: https://digitalcontentnext.org/wp-content/uploads/2018/08/DCN-Google-Data-Collection-Paper.pdf https://digitalcontentnext.org/wp-content/uploads/2018/08/DC...

econ4all 8y ago

It's interesting how the media cartel front thinks it's safe to drop all pretense and just post their opposition research like that. http://digitalcontentnext.org/membership/members http://digitalcontentnext.org/membership/members http://digitalcontentnext.org/about/overview/ http://digitalcontentnext.org/about/overview/ At the very least it reflects poorly on their member's coverage of FB/google.

TangoTrotFox 8y ago
I don't think it should matter who simple falsifiable statements of facts come from. 'Adversarial forces' would be the ones most motivated to discover these bits of information, and they're incredibly valuable. I mean this is an experiment anybody with a couple of phones and a dash of technical ability could carry out themselves. I would contrast this against the sort of macro-level research that involves modeling and other more subjective components subject to extensive 'tuning.' For instance the 'Uber drivers earn $3/hour' piece that ran a while back. The extensive modeling and obscured assumptions make it much easier to create fake conclusions that are very difficult to falsify.
- deleted 8y ago
  [deleted]
zepto 8y ago
Who cares? What matters is whether it is true or not. Are you just trying to smear the information with an ad hominem, or do you have reason to believe it is inaccurate?
- econ4all 8y ago
  It matters because the manner in which any "findings" are presented has a big role in shaping public opinion and in this case the opinion shapers are purpofuly distorting finding to attack an advesary. See the WSJ's false report that google let's developers read user emails, it's technically true but that only happens after the user allows it via an explicit permission screen. Or the AP's claim that google keeps track of user location ever after location history is turned off, which again is technically true but that is because location history is just a location feature and not the master switch for all location features. In both cases other media outlets repeated the claims without further commentery of research and this cartel might be the reason.

close04 8y ago

People will look fondly to the times when they thought Microsoft's telemetry was the worst privacy issue to hit them...

fartcannon 8y ago
Microsoft has not stopped spying, even now. They will likely be emboldened by their competitors exposure. Slowly boiling us all.
- close04 8y ago
  Nice way of missing the point. I don't expect MS will ever stop. There's no incentive, people already proved that there's no treatment so bad that they will put their foot down and demand a change. Google opened the way for collecting massive amounts of data by any means possible (including and not limited to tricking the user into thinking they aren't). And people seem pretty at ease with this, comparatively few are as angered by the revelations these days as they were about MS's relatively harmless telemetry. So going back, at some point you'll miss the times when you thought MS collecting data about your hardware and drivers was "the evilest evil".
  - fartcannon 8y ago
    It might just be me, but I believe most people thought Google was a worse violator prior to Windows 10. Some (probably naive) people used to think Windows was the last bastion of non-Linux hope.
    - close04 8y ago
      Not really. MS has (had) a very crappy image that they worked really hard to build during the early 2000s. That reputation stuck. And it doesn't help that most people are ignorant and are always willing to ignore an informed opinion over one that sounds cool. MS and Windows 10 are still far behind Google in the data collection game. Especially since they put that privacy wizard front and center at the first setup. But as I said, every second you spend defending Google and throwing shade at MS because it feels like the cool thing to do, Google adds another data collection item in their policy, hides yet another setting under 5 layers of options, and tricks you again into thinking that toggling a switch to off actually disables it. It's not like they did exactly that for 10 years now to the point where they are by far the largest private data collection effort in the history of man kind. And yes, I'm sure MS will follow suit. Why not? Google seems to be doing just fine.

throwawaymanbot 8y ago

Look how quick the comments thread on this research got Hijacked in to site performance..

gooftop 8y ago

In some ways not surprised, but its still a LOT of data (in sheer volume). Does Google even keep the data and analyze it, or is much of it thrown away as digital exhaust? Anyone know?

OnlyRepliesToBS 8y ago

looks class actiony

yuhong 8y ago

My favorite is trying to trace the problems back to Larry/Sergey, which is why I wrote the essay. Worth mentioning that one of the hardest parts of writing it was tracing the history of things like Google Analytics.

yuhong 8y ago
I forgot to mention that I had an Ask HN on this: https://news.ycombinator.com/item?id=17447280 https://news.ycombinator.com/item?id=17447280