13 ms·

Logs were our lifeblood, now they're our liability

188 points by vinnyglennon 7y ago

rubbingalcohol 7y ago
This brings up really good points. One of the best practices going forward is to minimize storage (and logging) of any personally identifiable information. Under GDPR, IP addresses can be considered PII, so it makes sense to set up an anonymizer for nginx ip address logs. There is a great Stack Overflow answer on this: https://stackoverflow.com/questions/6477239/anonymize-ip-logging-in-nginx https://stackoverflow.com/questions/6477239/anonymize-ip-log... But also there's some app hygeine involved. At least one of the recent "data breach" notifications involved not an actual leak of personal information, but unsanitized logs containing personal information that should not have been shared intra-organizationally. I forget the company that did this, but they notified as if it had been a breach even though passwords had just been logged internally. When testing it's convenient to do stuff like ``` console.log('username: ', req.body.username); console.log('password: ', req.body.password); ``` but it's all too easy to forget about it when you're working on a million things. So a big part of the solution is mindfulness (do I _really_ need to log this?)
- ehsankia 7y ago
  At least one? Almost every one of the recent "plaintext password" headlines, of which there has been half a dozen or more, were due to that. And I'm sure there are many more cases happening right now without being noticed. It definitely is big issue. And no, I don't think in most cases it's forgetting to remove the `console.log(req.body.password)`, but rather having a much wider `console.log(req)` which you didn't realize contains (or could in some other code path contain) a password. Or some log statement much deeper, 2-3 layers of abstractions away, logging some struct passing through the system, which happens to contain PII. It definitely isn't a trivial issue with a simple solution, as some people commenting on such headlines seem to imply.
  - ikiris 7y ago
    Its not that hard when you taint structs with PII, so that the log library faults if it tries to log it.
    - zbentley 7y ago
      Ah yes, just handle tainting/laundering all data that flows into a system which is hooked up to thousands or hundreds of thousands of external data sources. Not that hard at all. /s
      - ikiris 7y ago
        So your alternative is... do nothing?
        ehsankia 7y ago
        Of course if your whole system is written with this in mind, it's doable. But many code bases are huge and fixing them isn't trivial.
- pmoriarty 7y ago
  "Under GDPR, IP addresses can be considered PII, so it makes sense to set up an anonymizer for nginx ip address logs." One thing I wonder about is what you would do if, say, you have an abuser on your site that you need to ban due to behavior detected after the fact through a log file. If one needs their IP in order to ban them, but their IP is anonymized, what do you do?
  - DiabloD3 7y ago
    One way hash of the IP.
    - jacquesm 7y ago
      The search space is so small you can simply create a table with all of the hashes and use that to reverse the hash.
      - thaumasiotes 7y ago
        Mandatory IPv6 could finally be here! ;D
      - GhettoMaestro 7y ago
        You could always salt it.
        thaumasiotes 7y ago
        How would that address the problem? Salts don't make hashes any less reversible.
        majewsky 7y ago
        Salts are only useful when you already know which hash to check e.g. because the user supplied the username that picks which password hash to check. When matching an IP against a list of salted hashes, you need to hash the input with every possible salt to compare against all hashes in the list. So for performance reasons, it's probably not feasible to use more than one salt (or a small number of salts). Then it's again very possible to reverse the list of salted hashes because the search space is number of all IP addresses times number of salts used, which is way less than the number of all hashes.
- hyperman1 7y ago
  This: Under GDPR, IP addresses can be considered PII,so it makes sense to set up an anonymizer is wrong. First of all, PII is a legal definition from mostly the US. The GDPR talks about Personal Data, which is different. If any consultant is talking about GDPR and PII, he is confused, stop listening. Have fun reading the details here: https://gdpr-info.eu/art-4-gdpr/ https://gdpr-info.eu/art-4-gdpr/ Which doesn't matter all that much, because the next part of the line is way to broad: Is an IP adres Personal Data? Maybe, if you can use it to track an individual. But if you arent an ISP and don't actively try to identify anybody with an IP, stop worrying. Next question: Do you need it? In general, using them for keeping the website running is normal usage.So no need for consent. Using them for attack prevention might actually be an industry best practice, and then the GDPR requires you to keep them. Next question: If you need them, how to keep them safe? Throw logs away after a while, encrypt backups, etc... The GDPR has been hijacked by consulting companies to extract money from everybody, so they do their utmost best to sow paranoia with all kinds of weird urban myths. DOn't believe it. Basically, do the normal IT best practices and stop worrying.
joshmarinacci 7y ago
I was really hoping this would be about the lumber industry.
- crummy 7y ago
  Thanks to GDPR, we're no longer allowed to count the rings to reveal PII.
  - wpasc 7y ago
    Not to mention the crazy restrictions around acorns
    - bicknergseng 7y ago
      They're nuts.
      - deleted 7y ago
        [deleted]
- gumby 7y ago
  Theres a joke about functional tables of logs in here somewhere, one that unifies forestry, mathematics, programming, the bible, and DevOps. (And perhaps explains why my spell corrector camelcased "devops")
- mwyah 7y ago
  Hmm, reminds me that a loooong time ago I used a Linux distro that had a log viewer, and the icon of the log viewer was a pile of lumber logs.
- drewbug 7y ago
  > ‘log’ originally denoted a thin quadrant of wood loaded to float upright in the water, whence ‘ship's journal’ in which information derived from this device was recorded.
- Razengan 7y ago
ping_pong 7y ago
All this effort to distill user behavior and user intent from logs... why not just ask them????? I'm still waiting for the option to see the ads that I want to see. I want to see movie trailers, something that I rarely see now because I don't watch TV. I want to see new video games. I want to see new books, what sports event are going on, etc. Why not ask me? It's literally not rocket science here but billions are spent on machine learning, clicks, storing exabytes of data or more trying to figure this shit out. Just ask me for fuck's sake, I'm more than willing to watch ads in exchange for a useful service!
- hakfoo 7y ago
  The problem is there's a huge trust gap on both sides. The advertisers figure that self-reported data may be faulty. In some cases it could be-- either maliciously ("I'm 92 years old and spend $600 per month on my phone bill!") or by omission ("I hadn't even thought about it, but my lease is up in 3 months and high-yielding car ads might well be useful for me") Consumers won't be easily convinced they're being taken seriously-- if they have to go out of their way to customize ads, and then see equal or more ads than before, or non-laser-focused ads, their trust goes out the window. It's also going to be difficult to stand out in a sea of banners and say "we're the high quality ad you customized... right next to 32 click-your-state-for-mortgage-broker-lead-arbitrage banners." I suspect there may also be a third distrust-- between ad networks and advertisers. If you ran ads for a product only on a really targeted audience, the figures may not look compelling to buyers. It might look better to say "This $10k ad campaign landed in front of one million weakly targeted eyeballs... and generated 40 sales" than to say "We spent $10k on a super-premium media blitz that hit 200 hand-selected individuals and sold 45 units."
  - arthurcolle 7y ago
    If there’s really a trust issue on the offer side (advertisers) then why is YouTube asking survey questions about my demographics and related nonsense? I am a consumer and I don’t want the cognitive pollution associated with having images and sound shoved in my face. Full stop. Separately, worth mentioning, is the fact that the idea of introducing purposeful limitations deliberately to get people to pay more is also evil (I can play YouTube on my desktop and switch tabs, and on iOS for ex. it stops playback so you can buy YouTube Red). EDIT: Offer side/bid side is ambiguous since the model is “inverted” with advertising. I’m leaving my comment as is, but its probably better to say “bid side” in this specific case.
dxbydt 7y ago
The 5 user Nielsen test referred to in this article is quite inaccurate. If ( and that's a very big if) the users are IID, then yes, 5 users is all you need. But your users from Russia aren't the users from USA aren't the users from China etc. Even if your userbase belongs to single country, there's differences between CA user vs TX user vs NY user etc. Further, the analysis isn't static in time! So you as a single user will be a different person tomorrow because you are more familiar with the software, or your mood is better, or your worktable/mind is less cluttered so you can pay more attention etc. In other words, world isn't multinomial coins with fixed head probabilities. Here the coins are people & the probabilities change over time. So the only sufficiency statistics are order stats. Hence logs. Nielsen adds a massive caveat "The formula only holds for comparable users who will be using the site in fairly similar ways" - its possible to find such people in very homogenous groups. Like if you have a GRE saas & target all only the white college kids taking the GRE, hopefully 5 white kids is enough. Now you bring in black & brown & hispanic & chinese & so forth...maybe 5 of each. Or maybe you want to separate by sexes, so 10 of each...it gets complicated very soon, which is why its much simpler to just log everything/everybody.
- zer00eyz 7y ago
  > The 5 user Nielsen test referred to in this article is quite inaccurate > "The formula only holds for comparable users who will be using the site in fairly similar ways" This is the article where Neilson breaks down what is being tested, and why it is statistically relevant. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w... Neilson is looking to solve HCI(1) and Human Factors(2) issues - and most of these are byproducts of having to have a deep (insider) understanding of a product and that bubbling up into your UI. You are going to catch a lot of errors that fit the adage "can't see the forest for the trees". Having sat through a LOT Of these tests, you will pick out user frustrations, and reasons for product abandonment that would likely be NON apparent in a log. Your examples of US/China, TX vs CA and GRE with race and class MIGHT be relevant but it is going to depend a whole lot more on what your building. The problem is there are other means and places where these issues might manifest, and again use testing would tell you a lot. If we were to build a VR game that used a chopstick like interface, and test it only in china, we would likely think that we had a good product. If we find out later that "this isn't selling in America" then testing in that demographic group would quickly give us the insight that people lack the muscle memory to use this intuitively. There isn't any log in the known universe that would give us that clue, and "test here" can (and likely would) be gleaned by other means. When you get past HCI and Human factors log data can be useful, and be a contra-indicator of the results of formal testing. Given a choice between A and B in a formal setting may give you one set of results even with a large sample size, but real world behavior turns out to be very different. This is akin to people slowing down when they see a police car - but driving fast when one isn't present or kids acting differently because they know someone is watching. We aren't discussing UI and UI interactions were now discussing human behavior, and preference. I can't tell you how many times I have seen the non preferred solution be the winning one in an A/B test, but I would generally bet against what the group likes and pick the most garish solution as the winner. These behavioral types of tests can only really be driven by logs, by people being themselves and "feeling" unmonitored, and accurate demographic (to your point) slicing and sorting. En mass people are far more predictable than they would like to believe. Were delving into something more along the lines of Asimovs Psychohistory(3) as I don't think these sorts of statistically predicable behaviors have been given a formal name. 1. https://en.wikipedia.org/wiki/Human–computer_interaction https://en.wikipedia.org/wiki/Human–computer_interaction 2. https://en.wikipedia.org/wiki/Human_factors_and_ergonomics https://en.wikipedia.org/wiki/Human_factors_and_ergonomics 3. https://en.wikipedia.org/wiki/Psychohistory_(fictional) https://en.wikipedia.org/wiki/Psychohistory_(fictional)
EdwardDiego 7y ago
The author touches on the cargo cult belief that data has inherent value, so collect all of it, hire a data scientist or ML expert and point them at it, and watch the value flow! Seems to come from the same place as the belief that the Cloud magically makes everything resilient and scalable without any extra effort on your part. Just put it in the cloud, and then give your CTO a bonus for suggesting the cloud, and suddenly you don't need to worry about sysops. Don't get me wrong, "log all the things" is a good place to start when you need to figure out what's actually worth logging - but it needs to be followed by a rigorous prune. Otherwise your data-lake turns into a data-swamp, you collate a lot of noise that makes it harder to find signals, and people eventually end up spending a lot of time trying to figure out what's actually used, if any, when Hadoop gets full or the S3 bill gets too high.
deleted 7y ago
[deleted]
dredmorbius 7y ago
Addressing the value (and limits) of sampling: Yes, statistical sampling is a hugely useful practice, and is frequently used, at least by those who are familiar with its power and capabilities. Depending on what you can see, it may or may not be particularly useful. For activity logs, you are getting a bunch of relevant information, though if you stick to just sampling log records, you may miss useful information, such as paths through a site, session data, and the like. In doing analysis of the scale and scope of usage and activity of the late and unlamented Google+, I had the opportunity to sample based on profile IDs, which Google had helpfully stashed in a set of robots.txt sitemap files, back in 2015. More recently, when seeking information on the number, size, and activity of G+ Communities (effectively: groups), I could perform a similar sampling based on the group IDs, also provided via sitemaps. For a basic assessment of how many active users and groups there were, a small sample, as few as 100 or so IDs, selected at random, were sufficient to give a general feel. But there's a lot of variance hidden in 2 billion registered users (as of 2015), or the 8 million Communities existing as of January 2019. And for detailed measurement of the most active users and groups, a very small fraction of the total (0.1% of users, and the top 50 or so of 8 million communities, or 0.000625%), the releative sampling population wasn't the total user or group count, but that small subset, randomly distributed throughout the whole, comprising that sample of interest. To find the very most active users and groups, in other words, you have to sample a lot of datapoints. (Mind: if I'd had log data, they'd have fallen straight out of that. I didn't. Which is itself another lesson: in most cases you're interested in activity and not population as a primary analysis variable.) Given my tools and methods -- requesting URLs and scraping, from a desktop system over residential broadband -- there were limits to the amount of sampling I could do. 50,000 profiles were doable in a couple of days, but a larger pull would have scaled linearlly in time. For Communities, I did a largish pull based on a minimum level of resolution I thought would be useful, based on 12,000 (again, randomly selected) Communities. In the end I lucked out as a third party was able to provide a comprehensive dataset of all 8 million communities and summary metadata, from which I could validate my earlier sample-based methods. But yes, working with hundreds or thousands of records, rather than millions or billions, often makes sense, is useful, and requires vastly fewer resources (compute, time, bandwidth). For getting a rough idea of just

188 points by vinnyglennon 7y ago

This brings up really good points. One of the best practices going forward is to minimize storage (and logging) of any personally identifiable information. Under GDPR, IP addresses can be considered PII, so it makes sense to set up an anonymizer for nginx ip address logs. There is a great Stack Overflow answer on this: https://stackoverflow.com/questions/6477239/anonymize-ip-logging-in-nginx https://stackoverflow.com/questions/6477239/anonymize-ip-log... But also there's some app hygeine involved. At least one of the recent "data breach" notifications involved not an actual leak of personal information, but unsanitized logs containing personal information that should not have been shared intra-organizationally. I forget the company that did this, but they notified as if it had been a breach even though passwords had just been logged internally. When testing it's convenient to do stuff like ``` console.log('username: ', req.body.username); console.log('password: ', req.body.password); ``` but it's all too easy to forget about it when you're working on a million things. So a big part of the solution is mindfulness (do I _really_ need to log this?)

ehsankia 7y ago
At least one? Almost every one of the recent "plaintext password" headlines, of which there has been half a dozen or more, were due to that. And I'm sure there are many more cases happening right now without being noticed. It definitely is big issue. And no, I don't think in most cases it's forgetting to remove the `console.log(req.body.password)`, but rather having a much wider `console.log(req)` which you didn't realize contains (or could in some other code path contain) a password. Or some log statement much deeper, 2-3 layers of abstractions away, logging some struct passing through the system, which happens to contain PII. It definitely isn't a trivial issue with a simple solution, as some people commenting on such headlines seem to imply.
- ikiris 7y ago
  Its not that hard when you taint structs with PII, so that the log library faults if it tries to log it.
  - zbentley 7y ago
    Ah yes, just handle tainting/laundering all data that flows into a system which is hooked up to thousands or hundreds of thousands of external data sources. Not that hard at all. /s
    - ikiris 7y ago
      So your alternative is... do nothing?
      - ehsankia 7y ago
        Of course if your whole system is written with this in mind, it's doable. But many code bases are huge and fixing them isn't trivial.
pmoriarty 7y ago
"Under GDPR, IP addresses can be considered PII, so it makes sense to set up an anonymizer for nginx ip address logs." One thing I wonder about is what you would do if, say, you have an abuser on your site that you need to ban due to behavior detected after the fact through a log file. If one needs their IP in order to ban them, but their IP is anonymized, what do you do?
- DiabloD3 7y ago
  One way hash of the IP.
  - jacquesm 7y ago
    The search space is so small you can simply create a table with all of the hashes and use that to reverse the hash.
    - thaumasiotes 7y ago
      Mandatory IPv6 could finally be here! ;D
    - GhettoMaestro 7y ago
      You could always salt it.
      - thaumasiotes 7y ago
        How would that address the problem? Salts don't make hashes any less reversible.
      - majewsky 7y ago
        Salts are only useful when you already know which hash to check e.g. because the user supplied the username that picks which password hash to check. When matching an IP against a list of salted hashes, you need to hash the input with every possible salt to compare against all hashes in the list. So for performance reasons, it's probably not feasible to use more than one salt (or a small number of salts). Then it's again very possible to reverse the list of salted hashes because the search space is number of all IP addresses times number of salts used, which is way less than the number of all hashes.
hyperman1 7y ago
This: Under GDPR, IP addresses can be considered PII,so it makes sense to set up an anonymizer is wrong. First of all, PII is a legal definition from mostly the US. The GDPR talks about Personal Data, which is different. If any consultant is talking about GDPR and PII, he is confused, stop listening. Have fun reading the details here: https://gdpr-info.eu/art-4-gdpr/ https://gdpr-info.eu/art-4-gdpr/ Which doesn't matter all that much, because the next part of the line is way to broad: Is an IP adres Personal Data? Maybe, if you can use it to track an individual. But if you arent an ISP and don't actively try to identify anybody with an IP, stop worrying. Next question: Do you need it? In general, using them for keeping the website running is normal usage.So no need for consent. Using them for attack prevention might actually be an industry best practice, and then the GDPR requires you to keep them. Next question: If you need them, how to keep them safe? Throw logs away after a while, encrypt backups, etc... The GDPR has been hijacked by consulting companies to extract money from everybody, so they do their utmost best to sow paranoia with all kinds of weird urban myths. DOn't believe it. Basically, do the normal IT best practices and stop worrying.

joshmarinacci 7y ago

I was really hoping this would be about the lumber industry.

crummy 7y ago
Thanks to GDPR, we're no longer allowed to count the rings to reveal PII.
- wpasc 7y ago
  Not to mention the crazy restrictions around acorns
  - bicknergseng 7y ago
    They're nuts.
    - deleted 7y ago
      [deleted]
gumby 7y ago
Theres a joke about functional tables of logs in here somewhere, one that unifies forestry, mathematics, programming, the bible, and DevOps. (And perhaps explains why my spell corrector camelcased "devops")
mwyah 7y ago
Hmm, reminds me that a loooong time ago I used a Linux distro that had a log viewer, and the icon of the log viewer was a pile of lumber logs.
drewbug 7y ago
> ‘log’ originally denoted a thin quadrant of wood loaded to float upright in the water, whence ‘ship's journal’ in which information derived from this device was recorded.
Razengan 7y ago

ping_pong 7y ago

All this effort to distill user behavior and user intent from logs... why not just ask them????? I'm still waiting for the option to see the ads that I want to see. I want to see movie trailers, something that I rarely see now because I don't watch TV. I want to see new video games. I want to see new books, what sports event are going on, etc. Why not ask me? It's literally not rocket science here but billions are spent on machine learning, clicks, storing exabytes of data or more trying to figure this shit out. Just ask me for fuck's sake, I'm more than willing to watch ads in exchange for a useful service!

hakfoo 7y ago
The problem is there's a huge trust gap on both sides. The advertisers figure that self-reported data may be faulty. In some cases it could be-- either maliciously ("I'm 92 years old and spend $600 per month on my phone bill!") or by omission ("I hadn't even thought about it, but my lease is up in 3 months and high-yielding car ads might well be useful for me") Consumers won't be easily convinced they're being taken seriously-- if they have to go out of their way to customize ads, and then see equal or more ads than before, or non-laser-focused ads, their trust goes out the window. It's also going to be difficult to stand out in a sea of banners and say "we're the high quality ad you customized... right next to 32 click-your-state-for-mortgage-broker-lead-arbitrage banners." I suspect there may also be a third distrust-- between ad networks and advertisers. If you ran ads for a product only on a really targeted audience, the figures may not look compelling to buyers. It might look better to say "This $10k ad campaign landed in front of one million weakly targeted eyeballs... and generated 40 sales" than to say "We spent $10k on a super-premium media blitz that hit 200 hand-selected individuals and sold 45 units."
- arthurcolle 7y ago
  If there’s really a trust issue on the offer side (advertisers) then why is YouTube asking survey questions about my demographics and related nonsense? I am a consumer and I don’t want the cognitive pollution associated with having images and sound shoved in my face. Full stop. Separately, worth mentioning, is the fact that the idea of introducing purposeful limitations deliberately to get people to pay more is also evil (I can play YouTube on my desktop and switch tabs, and on iOS for ex. it stops playback so you can buy YouTube Red). EDIT: Offer side/bid side is ambiguous since the model is “inverted” with advertising. I’m leaving my comment as is, but its probably better to say “bid side” in this specific case.

dxbydt 7y ago

The 5 user Nielsen test referred to in this article is quite inaccurate. If ( and that's a very big if) the users are IID, then yes, 5 users is all you need. But your users from Russia aren't the users from USA aren't the users from China etc. Even if your userbase belongs to single country, there's differences between CA user vs TX user vs NY user etc. Further, the analysis isn't static in time! So you as a single user will be a different person tomorrow because you are more familiar with the software, or your mood is better, or your worktable/mind is less cluttered so you can pay more attention etc. In other words, world isn't multinomial coins with fixed head probabilities. Here the coins are people & the probabilities change over time. So the only sufficiency statistics are order stats. Hence logs. Nielsen adds a massive caveat "The formula only holds for comparable users who will be using the site in fairly similar ways" - its possible to find such people in very homogenous groups. Like if you have a GRE saas & target all only the white college kids taking the GRE, hopefully 5 white kids is enough. Now you bring in black & brown & hispanic & chinese & so forth...maybe 5 of each. Or maybe you want to separate by sexes, so 10 of each...it gets complicated very soon, which is why its much simpler to just log everything/everybody.

zer00eyz 7y ago
> The 5 user Nielsen test referred to in this article is quite inaccurate > "The formula only holds for comparable users who will be using the site in fairly similar ways" This is the article where Neilson breaks down what is being tested, and why it is statistically relevant. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w... Neilson is looking to solve HCI(1) and Human Factors(2) issues - and most of these are byproducts of having to have a deep (insider) understanding of a product and that bubbling up into your UI. You are going to catch a lot of errors that fit the adage "can't see the forest for the trees". Having sat through a LOT Of these tests, you will pick out user frustrations, and reasons for product abandonment that would likely be NON apparent in a log. Your examples of US/China, TX vs CA and GRE with race and class MIGHT be relevant but it is going to depend a whole lot more on what your building. The problem is there are other means and places where these issues might manifest, and again use testing would tell you a lot. If we were to build a VR game that used a chopstick like interface, and test it only in china, we would likely think that we had a good product. If we find out later that "this isn't selling in America" then testing in that demographic group would quickly give us the insight that people lack the muscle memory to use this intuitively. There isn't any log in the known universe that would give us that clue, and "test here" can (and likely would) be gleaned by other means. When you get past HCI and Human factors log data can be useful, and be a contra-indicator of the results of formal testing. Given a choice between A and B in a formal setting may give you one set of results even with a large sample size, but real world behavior turns out to be very different. This is akin to people slowing down when they see a police car - but driving fast when one isn't present or kids acting differently because they know someone is watching. We aren't discussing UI and UI interactions were now discussing human behavior, and preference. I can't tell you how many times I have seen the non preferred solution be the winning one in an A/B test, but I would generally bet against what the group likes and pick the most garish solution as the winner. These behavioral types of tests can only really be driven by logs, by people being themselves and "feeling" unmonitored, and accurate demographic (to your point) slicing and sorting. En mass people are far more predictable than they would like to believe. Were delving into something more along the lines of Asimovs Psychohistory(3) as I don't think these sorts of statistically predicable behaviors have been given a formal name. 1. https://en.wikipedia.org/wiki/Human–computer_interaction https://en.wikipedia.org/wiki/Human–computer_interaction 2. https://en.wikipedia.org/wiki/Human_factors_and_ergonomics https://en.wikipedia.org/wiki/Human_factors_and_ergonomics 3. https://en.wikipedia.org/wiki/Psychohistory_(fictional) https://en.wikipedia.org/wiki/Psychohistory_(fictional)

EdwardDiego 7y ago

The author touches on the cargo cult belief that data has inherent value, so collect all of it, hire a data scientist or ML expert and point them at it, and watch the value flow! Seems to come from the same place as the belief that the Cloud magically makes everything resilient and scalable without any extra effort on your part. Just put it in the cloud, and then give your CTO a bonus for suggesting the cloud, and suddenly you don't need to worry about sysops. Don't get me wrong, "log all the things" is a good place to start when you need to figure out what's actually worth logging - but it needs to be followed by a rigorous prune. Otherwise your data-lake turns into a data-swamp, you collate a lot of noise that makes it harder to find signals, and people eventually end up spending a lot of time trying to figure out what's actually used, if any, when Hadoop gets full or the S3 bill gets too high.

deleted 7y ago

[deleted]

dredmorbius 7y ago

Addressing the value (and limits) of sampling: Yes, statistical sampling is a hugely useful practice, and is frequently used, at least by those who are familiar with its power and capabilities. Depending on what you can see, it may or may not be particularly useful. For activity logs, you are getting a bunch of relevant information, though if you stick to just sampling log records, you may miss useful information, such as paths through a site, session data, and the like. In doing analysis of the scale and scope of usage and activity of the late and unlamented Google+, I had the opportunity to sample based on profile IDs, which Google had helpfully stashed in a set of robots.txt sitemap files, back in 2015. More recently, when seeking information on the number, size, and activity of G+ Communities (effectively: groups), I could perform a similar sampling based on the group IDs, also provided via sitemaps. For a basic assessment of how many active users and groups there were, a small sample, as few as 100 or so IDs, selected at random, were sufficient to give a general feel. But there's a lot of variance hidden in 2 billion registered users (as of 2015), or the 8 million Communities existing as of January 2019. And for detailed measurement of the most active users and groups, a very small fraction of the total (0.1% of users, and the top 50 or so of 8 million communities, or 0.000625%), the releative sampling population wasn't the total user or group count, but that small subset, randomly distributed throughout the whole, comprising that sample of interest. To find the very most active users and groups, in other words, you have to sample a lot of datapoints. (Mind: if I'd had log data, they'd have fallen straight out of that. I didn't. Which is itself another lesson: in most cases you're interested in activity and not population as a primary analysis variable.) Given my tools and methods -- requesting URLs and scraping, from a desktop system over residential broadband -- there were limits to the amount of sampling I could do. 50,000 profiles were doable in a couple of days, but a larger pull would have scaled linearlly in time. For Communities, I did a largish pull based on a minimum level of resolution I thought would be useful, based on 12,000 (again, randomly selected) Communities. In the end I lucked out as a third party was able to provide a comprehensive dataset of all 8 million communities and summary metadata, from which I could validate my earlier sample-based methods. But yes, working with hundreds or thousands of records, rather than millions or billions, often makes sense, is useful, and requires vastly fewer resources (compute, time, bandwidth). For getting a rough idea of just