4 ms·

Can’t think of a way to reduce this to guidelines. I’ve seen a bunch of different errors. Mostly it’s a bad understanding of the math, or a bad understanding of

Can’t think of a way to reduce this to guidelines. I’ve seen a bunch of different errors. Mostly it’s a bad understanding of the math, or a bad understanding of what you’d want to do with the collected data. Just one example that comes to mind: suppose you want average (mean) latency. So you record, for every minute, the average latency for requests during that minute. Now you want to calculate the average latency for an entire day… but it’s impossible. You can calculate the daily average of the per-minute averages, but you’re averaging over minutes, rather than averaging over requests. This is just a simple example, but you can see how a seemingly innocuous decision sabotaged your ability to run the query you want. Maybe my advice is “learn calculus”.

noodlenotes 6y ago
Average is actually one of the "nicer" summary statistics to work with because you can recalculate it over a different aggregation level if you kept the volume. It's statistics like median, percentile that you have to worry about. In my experience, people over-rely on averages just because they're easy to re-aggregate when they really should be using something else.
- klodolph 6y ago
  Yeah, median and percentiles are awful. You usually want them only at the very top-end, as the final summary of some system. Like, “page renders within 500ms for 90% of users”. So, well meaning engineers will collect percentiles on individual components in the system, with the idea that if you want percentiles at the end, just collect percentiles everywhere. You’re left with garbage data that can’t be aggregated in a way that makes any sense.
  - kqr 6y ago
    That's not at all true. Any meaningful aggregation requires storing the empirical distribution of data for individual components. A percentile histogram is an excellent approximation to this, when done properly.
    - hasmanean 6y ago
      Agreed. Histograms can be added and aggregated. You can always estimate the mean from a histogram. So if you compute a histogram of latency values every minute, you can add them up for successive minutes to get the distribution over larger intervals.
    - klodolph 6y ago
      I think you actually agree with what I'm saying... which is that discarding the histogram and e.g. just collecting the 90%ile will leave you unable to do the analysis you want.
      - kqr 6y ago
        That does sound like something I'd say. Well spotted!
anonymousDan 6y ago
Sorry dumb question, but does this (averaging the averages) only not work because you can't guarantee the latency is normally distributed?
- kqr 6y ago
  Latencies are patently not normally distributed. I could offer you 100:1 odds on that bet. But the problem with averaging averages is that it makes no sense mathematically. The average (well, arithmetic mean, to be technical) has a well-defined meaning and interpretation in terms of the original data points. When you start averaging averages (or dear Lord percentiles!) you lose the last glimmer of interoperability you had.
  - nonameiguess 6y ago
    They're Poisson distributed, by construction. Queuing service time is actually the go to example when first introducing the Poisson distribution in an intro to statistics class. No need to make a bet. Anything that can't be negative isn't normally distributed. The bigger issue here, aside from the fact that you may be taking a simple average when you need to be taking a weighted average to aggregate your individual point estimates, as others have pointed out, is that you can estimate a sufficiently symmetric Poisson distribution with a normal distribution, but response latency is not symmetric, specifically because time travel isn't possible and you can't ever get negative response times to balance out the tail events on the right side of the distribution. Tail events with respect to latency can only be bad, not good. They're a bet with a hard cap on winnings but potentially unbounded losses. As such, you really need to consider what the worst case is and the expected frequency of that worst case.
    - mjb 6y ago
      Service latencies are typically not Poisson distributed, partially because their queuing systems are non-trivial, but mostly because there's typically a lot more going on than just queues. For one great deep dive on the topic, check out https://drkp.net/papers/latency-socc14.pdf https://drkp.net/papers/latency-socc14.pdf

noodlenotes 6y ago

Average is actually one of the "nicer" summary statistics to work with because you can recalculate it over a different aggregation level if you kept the volume. It's statistics like median, percentile that you have to worry about. In my experience, people over-rely on averages just because they're easy to re-aggregate when they really should be using something else.

klodolph 6y ago
Yeah, median and percentiles are awful. You usually want them only at the very top-end, as the final summary of some system. Like, “page renders within 500ms for 90% of users”. So, well meaning engineers will collect percentiles on individual components in the system, with the idea that if you want percentiles at the end, just collect percentiles everywhere. You’re left with garbage data that can’t be aggregated in a way that makes any sense.
- kqr 6y ago
  That's not at all true. Any meaningful aggregation requires storing the empirical distribution of data for individual components. A percentile histogram is an excellent approximation to this, when done properly.
  - hasmanean 6y ago
    Agreed. Histograms can be added and aggregated. You can always estimate the mean from a histogram. So if you compute a histogram of latency values every minute, you can add them up for successive minutes to get the distribution over larger intervals.
  - klodolph 6y ago
    I think you actually agree with what I'm saying... which is that discarding the histogram and e.g. just collecting the 90%ile will leave you unable to do the analysis you want.
    - kqr 6y ago
      That does sound like something I'd say. Well spotted!

anonymousDan 6y ago

Sorry dumb question, but does this (averaging the averages) only not work because you can't guarantee the latency is normally distributed?

kqr 6y ago
Latencies are patently not normally distributed. I could offer you 100:1 odds on that bet. But the problem with averaging averages is that it makes no sense mathematically. The average (well, arithmetic mean, to be technical) has a well-defined meaning and interpretation in terms of the original data points. When you start averaging averages (or dear Lord percentiles!) you lose the last glimmer of interoperability you had.
- nonameiguess 6y ago
  They're Poisson distributed, by construction. Queuing service time is actually the go to example when first introducing the Poisson distribution in an intro to statistics class. No need to make a bet. Anything that can't be negative isn't normally distributed. The bigger issue here, aside from the fact that you may be taking a simple average when you need to be taking a weighted average to aggregate your individual point estimates, as others have pointed out, is that you can estimate a sufficiently symmetric Poisson distribution with a normal distribution, but response latency is not symmetric, specifically because time travel isn't possible and you can't ever get negative response times to balance out the tail events on the right side of the distribution. Tail events with respect to latency can only be bad, not good. They're a bet with a hard cap on winnings but potentially unbounded losses. As such, you really need to consider what the worst case is and the expected frequency of that worst case.
  - mjb 6y ago
    Service latencies are typically not Poisson distributed, partially because their queuing systems are non-trivial, but mostly because there's typically a lot more going on than just queues. For one great deep dive on the topic, check out https://drkp.net/papers/latency-socc14.pdf https://drkp.net/papers/latency-socc14.pdf