5 ms·

I strongly disagree with Zed on this one (at least part of his diatribe - he makes some good points about sample size/ramp-up/etc). I don't care what your mean

by smanek 17y ago

I strongly disagree with Zed on this one (at least part of his diatribe - he makes some good points about sample size/ramp-up/etc). I don't care what your mean or standard deviation are. If your performance isn't symmetrically distributed about the mean (and it probably isn't), standard deviation probably doesn't mean what you expect. I only look at performance in 99th percentile (or sometimes, 99.9th percentile). If 99% of responses are being served in 100ms on my webapp, I'm happy. I don't care if the mean is 5ms with a 20ms standard deviation. As long as the performance at the 99th (or 99.9th) percentile meets a level of 'acceptability' that I have predetermined, I'm satisfied. And you should be too. This is much easier to deal with than trying to do real math. Just make sure that no more than 1% (or 0.1%, as the case may be) of requests take longer than X. Pre-emptive answer: I usually get someone asking me why I don't demand acceptable performance at the 99.99999th percentile or at the slowest request. It's simply an issue of resources. To get that kind of performance, you can't use garbage collection, context switches become an issue, disk buffering even comes into play. You have to basically write a hard real-time (or pseudo real-time) app, and that level of effort isn't worth it for a consumer facing webapp. If I was writing code for a pacemaker or something (and please, dear god, no one ever let me do that) it damn sure better be hard real-time though.

Locke1689 17y ago
I'm pretty sure your system is useless without a statistically significant sample size.
- jerf 17y ago
  Sampling is not relevant to smanek's point, since he can look at all events. You don't need to sample when you have all the data. On the other hand, I guess this argues against smanek's points being relevant to Zed's point, since statistics aren't really involved for the same reason. You don't need stats to analyze your performance if you have all the performance data; just read it off. (Edit: I suppose his point that you shouldn't be using averages but just thresholds is still a statistical argument, against using the wrong tool like "average" on the grounds that you don't actually want the "average".)
  - lmkg 17y ago
    Even if you have all the data of your current run, that's still a sample (broadly speaking) of all the possible runs in the future. This is one of the core aspects of statistics, but it's so fundamental that it gets glossed over rather than getting drilled into people's head like they need. As a result, I have to tell people that when running a survey, total responses are more important than response rate.
viraptor 17y ago
I can't really agree with you. If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die. Not that bad? Ok - let's have an example: Facebook - one random user-page view == 63 requests (reported by firebug). That means 16 people generate >1000 requests. Even with 99.9% requests within your "acceptable range", you get every 16'th person either missing an element on a page due to some DB timeout (for example), or seeing an incomplete page for a not "acceptable" time (with 1/63'th probability of it being the main document, if we don't know which element fails). Is that really an acceptable behaviour? I don't believe that a requirement saying "the complete website loads in less than 1s" is affected by GC or context switches at all, tbh. If you actually tested the website with load of 10x the average expected one (over a couple of hours of random scripted walk with a natural write/read balance - testing you'd do on a service you really care about) and you still get random glitches with the expected load, then there's most likely something wrong on a higher level (disk seeks failing? shared hosting overloaded? network? etc. etc.)
- Retric 17y ago
  Taking more than 100ms is not the same things as "time-out and die". Performance is focused on how long something takes and it normally assumes that the system is not broken. It would take a horribly broken system to timeout 1% of requests. Assuming that 1% of your requests taking long enough the user might notice, but still working is generally acceptable performance for a webapp. So a reasonable goal might be 99% within 150ms and 99.95% don't fail.
  - viraptor 17y ago
    smanek wrote: "I don't care what your mean or standard deviation are.", which if taken seriously, means exactly that 1/1000th request can take an hour to finish and he doesn't care. Anything over 10s in real-life means a time-out in connection, or user refreshing, or going to the next page, or... Going to 99.95% not-failing connections doesn't help you much IMHO. It means that you've gone from every 16'th person having a problem to every 32'th person. It's not acceptable at all.
thaumaturgy 17y ago
> I strongly disagree with Zed on this one... I'm unsurprised. Programmers have a remarkable belief that they are experts in every subject, and if you tell them that there is something wrong with the way they argue, they will argue with you about it.
- nailer 17y ago
  This recalled to mind recently meeting a physicist who had some experience of programming at a party. The physicist repeatedly insisted (and explained to me with an authoritative tone) that Python 'has no VM as it is a scripting language'.
- scott_s 17y ago
  I'd rather discussions here avoid such broad and inflammatory generalizations.
zedshaw 17y ago
That'll work, if you have access to 100% of the timings for 100% of the requests and can get that information without having your measurements also modify the results. Otherwise, it's just easier to do samples and then collect meta-statistics (which are always normal). In that case you'll need to know error rates which require things like variance and std. deviation. Another technique is to go with statistical process control theory which mostly rejects confidence intervals and instead focuses on live sampling of processes to watch for outliers that need justification. With those you're looking at std. deviation as a measurement of the range of what's been happening vs. what's currently happening.
- DaniFong 17y ago
  Meta-statistics for website are typically normal? I think for one way flow through systems this has a tendency to be true (and so is likely to arise on most test harnesses), but I recall that multi-way routing systems with feedback effects (like, say, IP networks or highways) tend to have substantially non-gaussian congestion statistics: typically worse that gaussian, actually. Consider B. Huberman et. al. www.hpl.hp.com/research/scl/papers/InternetCongestion/InternetCongestion.pdf One problem with relying upon standard deviations when you are trying to design a reliable system is that the majority of the variance in many distributions is concentrated in rare events. This is particularly true with long-tails (e.g. a Pareto distribution with alpha <= 2). If you happen to have a system conforming nearly to such statistics, and you keep your eyes on the mean and the standard deviation alone, you'll end up severely underestimating the standard deviation (which might not even converge) and you could be designing a faulty system. Of course, the real problem, how can one gain confidence in one's statistical analysis. Not so easily, really. So if there's two things people who know about statistics, it's that statistics is critical, and statistics is hard.
  - gambling8nt 17y ago
    What Zed is saying when he notes that meta-statistics are normal is that, thanks to the central limit theorem, the average and standard deviation of data sets collected from the same underlying probability distribution (with convergent average and standard deviation) will tend to be normally distributed (in the limit approaching infinite sample size), even if the underlying system behavior is far from a normal distribution. In practice you work with finite sample sizes, so an underlying distribution sufficiently far from normal will result in a non-normal distribution of meta-statistics--but in most applications, these sort of pathological distributions are largely irrelevant. Take our example of looking at response time for loading a web page. There is some finite point (say, 10 sec) beyond which we no longer care how much longer it takes. So instead of considering the distribution of response times t, we consider the distribution of min(t, 10 sec). This distribution only has support over a finite interval, so its meta-statistics normalize rapidly as you increase the number of trials. Using this will under-report the actual standard deviation in the response time (which might, as you say, not even converge), since we've eliminated extremely low probability events with very high response time, but as a practical matter this is largely irrelevant--if these events are high enough probability for us to care we'll notice them anyway. The point of this exercise is not to perfectly ascertain the underlying distribution of t, it is to develop useful predictions for system behavior in practice.
tel 17y ago
Totally agree. This was obviously one of Zed's older, asshole dramatic posts and it doesn't read anywhere more clearly than him yelling about people failing to do statistics correctly and then blanket assuming normality on something pretty unlikely to be normal. Your method is a safe bet as long as you can collect "enough" data to be sure that you're covering all cases you claim to be covering. That being said, power analysis and data interpretation (ramp up, confounders, etc.) are all more difficult and important than this rant conveys. One R function is not your cure-all.
dspeyer 17y ago
This works if your request stream is homogeneous. If it isn't (and they usually aren't), you need to break it up by type and check the 99.9th percentile of each of them. Otherwise you could find that 99.9% of HTTP GETs are asynchronous AJAX polling and the requests that matter are slower.

by smanek 17y ago

Locke1689 17y ago

I'm pretty sure your system is useless without a statistically significant sample size.

jerf 17y ago
Sampling is not relevant to smanek's point, since he can look at all events. You don't need to sample when you have all the data. On the other hand, I guess this argues against smanek's points being relevant to Zed's point, since statistics aren't really involved for the same reason. You don't need stats to analyze your performance if you have all the performance data; just read it off. (Edit: I suppose his point that you shouldn't be using averages but just thresholds is still a statistical argument, against using the wrong tool like "average" on the grounds that you don't actually want the "average".)
- lmkg 17y ago
  Even if you have all the data of your current run, that's still a sample (broadly speaking) of all the possible runs in the future. This is one of the core aspects of statistics, but it's so fundamental that it gets glossed over rather than getting drilled into people's head like they need. As a result, I have to tell people that when running a survey, total responses are more important than response rate.

viraptor 17y ago

I can't really agree with you. If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die. Not that bad? Ok - let's have an example: Facebook - one random user-page view == 63 requests (reported by firebug). That means 16 people generate >1000 requests. Even with 99.9% requests within your "acceptable range", you get every 16'th person either missing an element on a page due to some DB timeout (for example), or seeing an incomplete page for a not "acceptable" time (with 1/63'th probability of it being the main document, if we don't know which element fails). Is that really an acceptable behaviour? I don't believe that a requirement saying "the complete website loads in less than 1s" is affected by GC or context switches at all, tbh. If you actually tested the website with load of 10x the average expected one (over a couple of hours of random scripted walk with a natural write/read balance - testing you'd do on a service you really care about) and you still get random glitches with the expected load, then there's most likely something wrong on a higher level (disk seeks failing? shared hosting overloaded? network? etc. etc.)

Retric 17y ago
Taking more than 100ms is not the same things as "time-out and die". Performance is focused on how long something takes and it normally assumes that the system is not broken. It would take a horribly broken system to timeout 1% of requests. Assuming that 1% of your requests taking long enough the user might notice, but still working is generally acceptable performance for a webapp. So a reasonable goal might be 99% within 150ms and 99.95% don't fail.
- viraptor 17y ago
  smanek wrote: "I don't care what your mean or standard deviation are.", which if taken seriously, means exactly that 1/1000th request can take an hour to finish and he doesn't care. Anything over 10s in real-life means a time-out in connection, or user refreshing, or going to the next page, or... Going to 99.95% not-failing connections doesn't help you much IMHO. It means that you've gone from every 16'th person having a problem to every 32'th person. It's not acceptable at all.

thaumaturgy 17y ago

> I strongly disagree with Zed on this one... I'm unsurprised. Programmers have a remarkable belief that they are experts in every subject, and if you tell them that there is something wrong with the way they argue, they will argue with you about it.

nailer 17y ago
This recalled to mind recently meeting a physicist who had some experience of programming at a party. The physicist repeatedly insisted (and explained to me with an authoritative tone) that Python 'has no VM as it is a scripting language'.
scott_s 17y ago
I'd rather discussions here avoid such broad and inflammatory generalizations.

zedshaw 17y ago

That'll work, if you have access to 100% of the timings for 100% of the requests and can get that information without having your measurements also modify the results. Otherwise, it's just easier to do samples and then collect meta-statistics (which are always normal). In that case you'll need to know error rates which require things like variance and std. deviation. Another technique is to go with statistical process control theory which mostly rejects confidence intervals and instead focuses on live sampling of processes to watch for outliers that need justification. With those you're looking at std. deviation as a measurement of the range of what's been happening vs. what's currently happening.

DaniFong 17y ago
Meta-statistics for website are typically normal? I think for one way flow through systems this has a tendency to be true (and so is likely to arise on most test harnesses), but I recall that multi-way routing systems with feedback effects (like, say, IP networks or highways) tend to have substantially non-gaussian congestion statistics: typically worse that gaussian, actually. Consider B. Huberman et. al. www.hpl.hp.com/research/scl/papers/InternetCongestion/InternetCongestion.pdf One problem with relying upon standard deviations when you are trying to design a reliable system is that the majority of the variance in many distributions is concentrated in rare events. This is particularly true with long-tails (e.g. a Pareto distribution with alpha <= 2). If you happen to have a system conforming nearly to such statistics, and you keep your eyes on the mean and the standard deviation alone, you'll end up severely underestimating the standard deviation (which might not even converge) and you could be designing a faulty system. Of course, the real problem, how can one gain confidence in one's statistical analysis. Not so easily, really. So if there's two things people who know about statistics, it's that statistics is critical, and statistics is hard.
- gambling8nt 17y ago
  What Zed is saying when he notes that meta-statistics are normal is that, thanks to the central limit theorem, the average and standard deviation of data sets collected from the same underlying probability distribution (with convergent average and standard deviation) will tend to be normally distributed (in the limit approaching infinite sample size), even if the underlying system behavior is far from a normal distribution. In practice you work with finite sample sizes, so an underlying distribution sufficiently far from normal will result in a non-normal distribution of meta-statistics--but in most applications, these sort of pathological distributions are largely irrelevant. Take our example of looking at response time for loading a web page. There is some finite point (say, 10 sec) beyond which we no longer care how much longer it takes. So instead of considering the distribution of response times t, we consider the distribution of min(t, 10 sec). This distribution only has support over a finite interval, so its meta-statistics normalize rapidly as you increase the number of trials. Using this will under-report the actual standard deviation in the response time (which might, as you say, not even converge), since we've eliminated extremely low probability events with very high response time, but as a practical matter this is largely irrelevant--if these events are high enough probability for us to care we'll notice them anyway. The point of this exercise is not to perfectly ascertain the underlying distribution of t, it is to develop useful predictions for system behavior in practice.

tel 17y ago

Totally agree. This was obviously one of Zed's older, asshole dramatic posts and it doesn't read anywhere more clearly than him yelling about people failing to do statistics correctly and then blanket assuming normality on something pretty unlikely to be normal. Your method is a safe bet as long as you can collect "enough" data to be sure that you're covering all cases you claim to be covering. That being said, power analysis and data interpretation (ramp up, confounders, etc.) are all more difficult and important than this rant conveys. One R function is not your cure-all.

dspeyer 17y ago

This works if your request stream is homogeneous. If it isn't (and they usually aren't), you need to break it up by type and check the 99.9th percentile of each of them. Otherwise you could find that 99.9% of HTTP GETs are asynchronous AJAX polling and the requests that matter are slower.