3 ms·

It seems concerning that they would be hanging onto traffic logs for that long. Seems like a massive PII / data breach liability.

tutfbhuf 6y ago
Agreed, it's questionable why a company needs to save raw traffic logs for years.
- valuearb 6y ago
  Yea, it seems like this project should have started with a cost/benefit analysis. Like what is this data used for, who is using it, how often and how to quantify the benefits the users derive from it. Not sure which the data couldn’t just be compressed into daily summaries in the DB, while the raw logs are stored (Compressed) in the file system in case you ever had to go back.
- dt3ft 6y ago
  Came here for this comment. 4 years of traffic logs... is just plain brutal.
- bryan_w 6y ago
  You say that but people were pretty upset that Google couldn't confirm if anybody exploited the G+ security bug past the last two weeks that it was discovered.
  - m3047 6y ago
    +1 for them. Pretty cynical, I know. But imagine the request for legal discovery...
- deleted 6y ago
  [deleted]
dsr_ 6y ago
Right here at the beginning: "Our HAProxy1 logs aka traffic logs, are currently stored on two SQL Servers, one in New York, and one in Colorado. While we store a minimal summary of the traffic data, we have a lot of it. At the beginning of 2019, we had about 4.5 years of data, totaling about 38TB. " and I asked myself, why the hell would you want this data? Logs serve two purposes: - find a fault and fix it - audit what someone (or many someones) did You never need years of data to find evidence of a fault: either it's available in the last few weeks or you didn't log the data in the first place (unless, of course, it's a bug that only shows up on leapdays, or leapseconds, or is similarly rare...) So that leaves tracking your users in a way which isn't covered by your user's own history. Again, a month or two might be helpful, sometimes... but why years?
- Shog9 6y ago
  > unless, of course it's a bug that ... is similarly rare... I'll give an example: a bug in the revision ordering logic that could be triggered by two edits being submitted almost simultaneously where one of the editors had made another edit within the past few minutes. The bug resulted in rows being interleaved between the two revisions and the post being rendered as a combination of two unrelated edits. Almost impossible to reproduce. Affected a low double-digit number of posts out of millions edited over the course of several years. Was able to finally nail down the scenario by pulling logs for all editors involved in each of the events over the years and identifying the common factor. (You can argue this is a crazy way of storing revisions, but that's beside the point; at this scale even very unlikely bugs do show up with some frequency and it is nice to be able to investigate them)
  - buerkle 6y ago
    "at this scale even very unlikely bugs do show up with some frequency" exactly! In the past I've worked on systems where "this will never happen" occurs because "this will never happen" really means the probability of this occurring is extremely low. But when you're dealing with millions of possibilities that low probability event starts occurring every day. And when it occurs for one of your largest customers it's no longer "this will never happen" but "why is this happening?"
jpalomaki 6y ago
Stack Overflow HAproxy logs likely don’t have too much sensitive information. These are most likely logs about visited pages with timestamps - maybe with client IP addresses. Storing them raw can make sense. You might later come up with new ways to analyze the data which your aggregation logic of choice does not support.
- rrdharan 6y ago
  > maybe with client IP addresses https://www.whitecase.com/publications/alert/court-confirms-ip-addresses-are-personal-data-some-cases https://www.whitecase.com/publications/alert/court-confirms-...

tutfbhuf 6y ago

Agreed, it's questionable why a company needs to save raw traffic logs for years.

valuearb 6y ago
Yea, it seems like this project should have started with a cost/benefit analysis. Like what is this data used for, who is using it, how often and how to quantify the benefits the users derive from it. Not sure which the data couldn’t just be compressed into daily summaries in the DB, while the raw logs are stored (Compressed) in the file system in case you ever had to go back.
dt3ft 6y ago
Came here for this comment. 4 years of traffic logs... is just plain brutal.
bryan_w 6y ago
You say that but people were pretty upset that Google couldn't confirm if anybody exploited the G+ security bug past the last two weeks that it was discovered.
- m3047 6y ago
  +1 for them. Pretty cynical, I know. But imagine the request for legal discovery...
deleted 6y ago
[deleted]

dsr_ 6y ago

Right here at the beginning: "Our HAProxy1 logs aka traffic logs, are currently stored on two SQL Servers, one in New York, and one in Colorado. While we store a minimal summary of the traffic data, we have a lot of it. At the beginning of 2019, we had about 4.5 years of data, totaling about 38TB. " and I asked myself, why the hell would you want this data? Logs serve two purposes: - find a fault and fix it - audit what someone (or many someones) did You never need years of data to find evidence of a fault: either it's available in the last few weeks or you didn't log the data in the first place (unless, of course, it's a bug that only shows up on leapdays, or leapseconds, or is similarly rare...) So that leaves tracking your users in a way which isn't covered by your user's own history. Again, a month or two might be helpful, sometimes... but why years?

Shog9 6y ago
> unless, of course it's a bug that ... is similarly rare... I'll give an example: a bug in the revision ordering logic that could be triggered by two edits being submitted almost simultaneously where one of the editors had made another edit within the past few minutes. The bug resulted in rows being interleaved between the two revisions and the post being rendered as a combination of two unrelated edits. Almost impossible to reproduce. Affected a low double-digit number of posts out of millions edited over the course of several years. Was able to finally nail down the scenario by pulling logs for all editors involved in each of the events over the years and identifying the common factor. (You can argue this is a crazy way of storing revisions, but that's beside the point; at this scale even very unlikely bugs do show up with some frequency and it is nice to be able to investigate them)
- buerkle 6y ago
  "at this scale even very unlikely bugs do show up with some frequency" exactly! In the past I've worked on systems where "this will never happen" occurs because "this will never happen" really means the probability of this occurring is extremely low. But when you're dealing with millions of possibilities that low probability event starts occurring every day. And when it occurs for one of your largest customers it's no longer "this will never happen" but "why is this happening?"

jpalomaki 6y ago

Stack Overflow HAproxy logs likely don’t have too much sensitive information. These are most likely logs about visited pages with timestamps - maybe with client IP addresses. Storing them raw can make sense. You might later come up with new ways to analyze the data which your aggregation logic of choice does not support.

rrdharan 6y ago
> maybe with client IP addresses https://www.whitecase.com/publications/alert/court-confirms-ip-addresses-are-personal-data-some-cases https://www.whitecase.com/publications/alert/court-confirms-...