5 ms·

You know what I've always found really useful? Entire distributions. Forget means, medians, percentiles, etc. — give me the complete distribution (along with

You know what I've always found really useful? Entire distributions. Forget means, medians, percentiles, etc. — give me the complete distribution (along with sample size) so I can understand all of the nuances of the data. (Better yet, just give me the raw data so I can analyze it myself. I find it hard to blindly trust someone else's conclusions considering all of the p-hacking going on nowadays.)

secabeen 11y ago
Yeah. We're starting to contemplate a new monitoring/performance/etc. system, and I'm thinking I don't want to do any of the long-term storage aggregration like in rrd. We'll store the full data for each item forever. Not sure if we can afford the storage, but I want to try for a while and see if it works.
- jedberg 11y ago
  We did something similar at Netflix. We had all the aggregations but also stored all the raw data. The raw data would be pushed out of the live system and replaced with aggregates and then stored as flat text in S3. If for some reasons you needed the old data, you just put in a ticket with the monitoring team to load the data back into the live system (I think this is even self service now). The system would then load the data from S3 back into the live system via Hadoop. Turns out it was pretty cheap to store highly compressible files in S3.
  - halayli 11y ago
    I've built a similar system as well. Raw data compressed and stored in s3, but aggregations are stored in postgres. Data stored in Postgres is compressed binary representation of a histogram and I added few C functions in postgres to do things like select x,y, histogram_percentile(histogram_merge(data), 99) group by x,y etc..
- amluto 11y ago
  My company stores all raw data for pretty much everything forever (and, in finance, there are a lot of things). It's binary, compressed with xz, and stored in several places. The grand total Glacier cost is something like $50/month. (Do NOT, for the love of your sanity, emulate this design exactly. Use Backblaze or Google's nearline storage. Do not touch Glacier at all if you can avoid it. When I wrote the Glacier integration, I did it because it was far cheaper than its competitors. That's no longer the case.)
robzyb 11y ago
Funnily enough, I am writing an article about this now. FYI I don't think its right to say "forget means, medians, percentiles, etc." They are hugely useful. You just have to note that they aren't everything. As you've noted, only the raw data is everything.
- edanm 11y ago
  But the raw data isn't everything either, it's usually just a sample. The analysis of it is there to explain what the data means. If you're saying that people should provide the full data along with their analysis, I fully agree! But if you're saying "skip the analysis", then unfortunately I, and most people, are not statistically educated enough to do all the analysis ourselves.
srean 11y ago
Some caution is warranted. Always keep in mind that estimating distributions from samples is very hard, in particular for multidimensional and continuous variables. Sometimes even infinite samples do not suffice (I am not kidding). The good news is that there are interesting and relevant tasks (for example classification) where you do not need to know the distribution to solve the task optimally, although if you somehow managed to do so (find the distribution) you would be able to solve the task optimally. Or more succinctly, knowing the distribution is sufficient but not necessary for such tasks (small mercies)
cm2187 11y ago
A nice distribution diagram is very informative but if you want to track the performance through time and have warnings triggers in place you need to be able to capture what you care about in a handful of numeric metrics.
redcalx 11y ago
It's also often useful to think of the true distribution of the underlying system/dynamic in question, and to consider a histogram as a sample that therefore has sample error. E.g. if the histogram's tail bins have very few samples then they're giving you some information about the shape of the distribution in that region, but not the exact probability of events in that region.
- xaprb 11y ago
  This is an excellent point. The distribution of the measurements are the distribution of the sample, not of the population. In my experience very few non-statisticians understand the difference. Statistics is hard.
  - Xcelerate 11y ago
    This is true. I almost mentioned the difficulties of estimating the kernel density bandwidth in my original post, but thought that would be getting too far on a tangent.
GFK_of_xmaspast 11y ago
Your post is a little confused. For actual data, there's no distinction between the empirical distribution and the data, so when you want "along with sample size" with your "distribution", that's what you're getting. (same with your "raw data").

secabeen 11y ago

Yeah. We're starting to contemplate a new monitoring/performance/etc. system, and I'm thinking I don't want to do any of the long-term storage aggregration like in rrd. We'll store the full data for each item forever. Not sure if we can afford the storage, but I want to try for a while and see if it works.

jedberg 11y ago
We did something similar at Netflix. We had all the aggregations but also stored all the raw data. The raw data would be pushed out of the live system and replaced with aggregates and then stored as flat text in S3. If for some reasons you needed the old data, you just put in a ticket with the monitoring team to load the data back into the live system (I think this is even self service now). The system would then load the data from S3 back into the live system via Hadoop. Turns out it was pretty cheap to store highly compressible files in S3.
- halayli 11y ago
  I've built a similar system as well. Raw data compressed and stored in s3, but aggregations are stored in postgres. Data stored in Postgres is compressed binary representation of a histogram and I added few C functions in postgres to do things like select x,y, histogram_percentile(histogram_merge(data), 99) group by x,y etc..
amluto 11y ago
My company stores all raw data for pretty much everything forever (and, in finance, there are a lot of things). It's binary, compressed with xz, and stored in several places. The grand total Glacier cost is something like $50/month. (Do NOT, for the love of your sanity, emulate this design exactly. Use Backblaze or Google's nearline storage. Do not touch Glacier at all if you can avoid it. When I wrote the Glacier integration, I did it because it was far cheaper than its competitors. That's no longer the case.)

robzyb 11y ago

Funnily enough, I am writing an article about this now. FYI I don't think its right to say "forget means, medians, percentiles, etc." They are hugely useful. You just have to note that they aren't everything. As you've noted, only the raw data is everything.

edanm 11y ago
But the raw data isn't everything either, it's usually just a sample. The analysis of it is there to explain what the data means. If you're saying that people should provide the full data along with their analysis, I fully agree! But if you're saying "skip the analysis", then unfortunately I, and most people, are not statistically educated enough to do all the analysis ourselves.

srean 11y ago

Some caution is warranted. Always keep in mind that estimating distributions from samples is very hard, in particular for multidimensional and continuous variables. Sometimes even infinite samples do not suffice (I am not kidding). The good news is that there are interesting and relevant tasks (for example classification) where you do not need to know the distribution to solve the task optimally, although if you somehow managed to do so (find the distribution) you would be able to solve the task optimally. Or more succinctly, knowing the distribution is sufficient but not necessary for such tasks (small mercies)

cm2187 11y ago

A nice distribution diagram is very informative but if you want to track the performance through time and have warnings triggers in place you need to be able to capture what you care about in a handful of numeric metrics.

redcalx 11y ago

It's also often useful to think of the true distribution of the underlying system/dynamic in question, and to consider a histogram as a sample that therefore has sample error. E.g. if the histogram's tail bins have very few samples then they're giving you some information about the shape of the distribution in that region, but not the exact probability of events in that region.

xaprb 11y ago
This is an excellent point. The distribution of the measurements are the distribution of the sample, not of the population. In my experience very few non-statisticians understand the difference. Statistics is hard.
- Xcelerate 11y ago
  This is true. I almost mentioned the difficulties of estimating the kernel density bandwidth in my original post, but thought that would be getting too far on a tangent.

GFK_of_xmaspast 11y ago

Your post is a little confused. For actual data, there's no distinction between the empirical distribution and the data, so when you want "along with sample size" with your "distribution", that's what you're getting. (same with your "raw data").