4 ms·

Yes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the b

Yes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the benefit of a new treatment gave a much better understanding. Some thought it clouded the issue. For example, when a new treatment caused a 1% "improvement", but the confidence interval extended from -10% to 10%, it was clear that the experiment didn't tell us how that metric was affected. This makes the decision feel more arbitrary. But that is exactly the point - the decision is arbitrary in that case, and the confidence interval tells us that, allowing us to focus on other trade-offs involved. If the confidence interval is 0.9% to 1.1%, we know that we can be much more confident in the effect. A big problem with this is that meaningful error bars can be extremely difficult to come by in some cases. For example, imagine having something like that for every prediction made by an ML model. I would love to have that, but I'm not aware of any reasonable way to achieve it for most types of models. The same goes for online experiments where a complicated experiment design is required because there isn't a way to do random allocation that results in sufficiently independent cohorts. On a similar note, regularly look at histograms (i.e., statistical distributions) for all important metrics. In one case, we were having speed issues in calls to a large web service. Many calls were completing in < 50 ms, but too many were tripping our 500 ms timeout. At the same time, we had noticed the emergence of two clear peaks in the speed histogram (i.e., it was a multimodal distribution). That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users. That knowledge allowed us to ignore wide swaths of code and spot the speed issues in some recently pushed personalization code that we might not have suspected otherwise.

kqr 3y ago
> This makes the decision feel more arbitrary. This is something I've started noticing more and more with experience: people really hate arbitrary decisions. People go to surprising lengths to add legitimacy to arbitrary decisions. Sometimes it takes the shape of statistical models that produce noise that is then paraded as signal. Often it comes from pseudo-experts who don't really have the methods and feedback loops to know what they are doing but they have a socially cultivated air of expertise so they can lend decisions legitimacy. (They used to be called witch-doctors, priests or astrologers, now they are management consultants and macroeconomists.) Me? I prefer to be explicit about what's going on and literally toss a coin. That is not the strategy to get big piles of shiny rocks though.
kqr 3y ago
> That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users. This is extremely common and one of the core ideas of statistical process control[1]. Sometimes you have just the one process generating values that are sort of similarly distributed. That's a nice situation because it lets you use all sorts of statistical tools for planning, inferences, etc. Then frequently what you have is really two or more interleaved processes masquerading as one. These distributions generate values that within each are sort of similarly distributed, but any analysis you do on the aggregate is going to be confused. Knowing the major components of the pretend-single process you're looking at puts you ahead of your competition -- always. [1]: https://two-wrongs.com/statistical-process-control-a-practitioners-guide.html https://two-wrongs.com/statistical-process-control-a-practit...

kqr 3y ago

> This makes the decision feel more arbitrary. This is something I've started noticing more and more with experience: people really hate arbitrary decisions. People go to surprising lengths to add legitimacy to arbitrary decisions. Sometimes it takes the shape of statistical models that produce noise that is then paraded as signal. Often it comes from pseudo-experts who don't really have the methods and feedback loops to know what they are doing but they have a socially cultivated air of expertise so they can lend decisions legitimacy. (They used to be called witch-doctors, priests or astrologers, now they are management consultants and macroeconomists.) Me? I prefer to be explicit about what's going on and literally toss a coin. That is not the strategy to get big piles of shiny rocks though.

kqr 3y ago

> That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users. This is extremely common and one of the core ideas of statistical process control[1]. Sometimes you have just the one process generating values that are sort of similarly distributed. That's a nice situation because it lets you use all sorts of statistical tools for planning, inferences, etc. Then frequently what you have is really two or more interleaved processes masquerading as one. These distributions generate values that within each are sort of similarly distributed, but any analysis you do on the aggregate is going to be confused. Knowing the major components of the pretend-single process you're looking at puts you ahead of your competition -- always. [1]: https://two-wrongs.com/statistical-process-control-a-practitioners-guide.html https://two-wrongs.com/statistical-process-control-a-practit...