6 ms·
The Low Base Rate Problem (2014)
- H8crilA 5y agoBTW, asking the question "what is the difference between power and significance/p-value" is a good litmus test for detecting insufficient competence in statistics. As the author puts it, unless you know this stuff well you shouldn't really be juding the "scientific" numbers. In fact you're probably (probably!) better off building a good intuition within the domain and then improvising the tests, in the sense that this way you're less likely to arrive at bogus conclusions, as opposed to someone who thinks they'll just apply some formula like this was a high school maths problem - here we plug in some magic that speaks back the truth. I've seen it time and again that (especially senior) people can rapidly and accurately judge the results of things based on very "un-scientific" improvised methods and numbers. YMMV.
- sokoloff 5y ago/me quickly googling “what is the difference between power and significance/p-value?” because I know I lack competence in statistics.
- tel 5y agoBroadly, there are two kinds of error: false positives and false negatives. A false positive arises because reality conspired to produce what you consider to be an anomalous result. False negatives arise because your detector wasn't sensitive enough to notice the anomalous result. The standard statistical practice is to fix your risk of false positives through the preemptive choice of a p-value threshold and then to attempt to minimize your risk of false negatives through increases in the "power" of your experiment. But, honestly, these two are in tension with one another. There are lots of possible choices. It's important to know the difference, though. It changes how you interpret results. A positive result might be doubted because you, as a reader, would have preferred more protection against false positives, and, thusly, a smaller p-value threshold. A negative result might be doubted because you believe that the experimenter under-powered their experiment. You might also, in a repeating experimental process, wish to obtain greater power by sacrificing false-positive protection. This is as easy as changing the p-value threshold. A last perspective is that in the typical scientific practice, you basically control false-positives through pushing the p-value down and making the experiment more challenging (reducing power). Then, you re-establish power through more expensive experiments (use better tools, run the experiment for longer, improve controls, improve data efficiency). Often the "typical" p-value is set by the community in relation to what level of false-positive the overall community is willing to tolerate in publication. That's why psychologists are happy with a 5% threshold whereas physicists might seek a threshold of 10^-3 or smaller. The physicists are upholding themselves to a much more expensive experimental standard and would vastly prefer false negatives.
- clircle 5y agoPower is a pre-test measure, and a p-value is a post test measure.
- leephillips 5y agoAll you have to do is ask an author of a paper in medicine or psychology that quotes a p-value to define p-value for you. That will open your eyes if you’re not already clued in.
- ZeroGravitas 5y agoI don't know if it's my civic-minded love for humanity or my laziness, but whenever someone describes a common failure mode and then expects me to do work to catch it every time it inevitably happens, my brain screams "no!". The tooling, processes and community should make this problem impossible by design and/or easily caught in a semi-automated manner.
- xpe 5y agoI hear you, and our perceptual lenses are part of our tooling. Our mental tooling. A change in mindset can be brought can be encouraged and developed by a combination of process and community. This includes educational processes supported by public policy, market mechanisms, and communities. I believe we need a revolution on how we educate ourselves. How we get there is non-obvious. If sound statistical thinking was taught, nurtured, and synthesized across many domains, I think it is likely that humanity would be better off.
- marginalia_nu 5y agoGood science just doesn't reduce to a script. You need to actually understand what you are doing to interpret the results correctly. It's why people spend a decade getting a PhD, and many years more still to become a professor. You can't do good science by putting numbers in a spreadsheet and have the computer do all the thinking for you.
- xpe 5y agoI missed the connection between your comment and its parent comment. Would you please elaborate? Perhaps you are thinking about a situation where the result of an AB test is accepted and incorporated into a product without reflection on the associated factors and context?
- marginalia_nu 5y agoI took the GP to mean they wanted tooling to do design and interpret A/B tests for them rather than having to consider cases like false negatives and what have you, to reduce them down to something that's easier to understand. My point is that the process of designing experiments and interpreting their outcome is nothing other than science. Science is hard and goes much deeper than the trivial stuff like significance and power. Even scientists, who know all these formulae inside and out struggle with constructing experiments and interpreting the outcome. To properly think about experimental outcomes, you need to actually have a firm grasp of statistics. The moment you try to convert it into a message that doesn't require understanding statistics is the moment it stops being informative.
- xpe 5y agoHere is a one page visual diagram (GPL licensed) I created a while back to show a large number of statistical classification metrics in a way that reveals an underlying symmetry. It also shows synonyms for metric across fields. https://github.com/xpe/statistical-classification-metrics/blob/master/statistical-classification-metrics.png https://github.com/xpe/statistical-classification-metrics/bl... (PDF and OmniGraffle versions also available)
- demopathos 5y agoWhat I don't understand is why power would be so relevant. I want to know if going from A to B would increase my revenue. I run an A/B test and see statistical significance, even if a minor one. I now know that B is better than A. I suppose the need for a power calculation comes in when considering effort. If I need 10 engineers for a month to build out a feature that won't get the power it needs for a year, it may not be worth it.
- xpe 5y agoNot just effort: there usually are many costs associated with change. Perhaps user disorientation, downtime, as well as a lack of long term understanding of how the change plays out in combination with other factors. A small improvement may not be worth it right now. Perhaps it can be deferred. Perhaps it makes sense to bundle it with other changes later. * * Some changes might reflect better on a product when they are rolled out together; e.g. to signal a major release.
- bkfunk 5y agoPower is important if you want to distinguish between noise and true signals. In other words, if you care about P(real effect | significant). From good ol' Bayes we get: P(real|sig) = P(sig|real) x P(real) / P(sig) P(sig|real) is the power; so if you have more power, all other things being equal (a bit of a weaselly caveat), the likelihood that your stat sig result is real is higher.
- thelamest 5y ago>I run an A/B test and see statistical significance That’s exactly what power gives you: a fighting chance to detect anything. Not sure where the misunderstanding comes from – it’s about the worthwhileness of tests themselves, not “features”. You may think about power visually by relating them to confidence intervals – higher power → more precise (expected) estimate. Low power → bands so wide that you can just as well use an RNG.
- placidpanda 5y ago> What I don't understand is why power would be so relevant. Doing the A/B test itself has a cost greater than just building the feature and releasing it (supporting two variants in production), and beyond that you also need to take into account the cost of acting on the results (i.e. if control wins what do you do? if it's a tie what do you do? best to budget for maximum possible effort, or the expected effort -- but expecting for the variant to win handily is budgeting for the minimum possible effort). I've seen multiple businesses that always schedule around shipping an A/B test and context switching to the next project while the results stream in. Any result that isn't shipping the variant after x weeks is a huge inconvenience that throws off multiple teams, which means all those cognitive biases start to creep in and make it comfortable to declare loser variants as wins or ties. While it's easy to write this behavior off as yet another way that groups make irrational decisions, I think the bit of truth in there is that sometimes, the cost of running the strictest, science-iest A/B test is simply too high. Power is a key part of how you reason that out up front, so you can make a rational decision not to test, or to modify your test to make the payoff worth it. For example: * Let's set the goal metric for something higher up the funnel which is further from our true goal (more $) but happens much more often, so we can see the effect in 1 week instead of 2 months * We really need to do this for [variety of business strategical decisions], so let's structure our experiment to make sure it won't cost us more than $X in a worst case scenario and find out in a few days rather than wait 2 months > I now know that B is better than A. You know that B outperformed A in the experiment. Checking statistical significance is like asking a trustworthy person "Are you sure?" and them saying "Yeah, I'm pretty sure". It's a percentage because it's sometimes wrong, and this doesn't account for the massive amount of real world factors that can still mean an experiment conducted with bulletproof math behind the analysis is still taking people down the wrong path.
- xpe 5y ago> The situation is even more depressing at lower conversion rates. Depressing? That is an unnecessary and unfortunate word choice. I would suggest 'pronounced'. The degree to which interventions can be distinguished is not inherently good or bad. To make such an assessment, context is required.
- placidpanda 5y agoRight you are, but in defense of the author, understanding this concept while looking through readouts done by past colleagues has also made me feel depressed at times.
- tqi 5y agoWhile this is all true, I think the reason why people so often run underpowered tests is not because they "lacks a firm understanding of statistical power" but because we have not made a compelling case for an alternative. UXR surveys? In my experience that is even more underpowered than AB tests. User testing? Often just as subjective as gut calls. Gut calls (ie product sense) seem like the only real choice, but I think there really needs to be an argument made for WHY that is the way, rather than just against other options.
- xpe 5y agoI tend to notice when people write 'the only choice' or 'the only real choice' when they probably mean something more like (1) an option that, when examined, seems best; or (2) an unexamined (sometimes dogmatic) choice. In some cases, a person may have a third intent: by linguistically limiting a choice set, they can frame or limit a discussion (for good or ill).
- tqi 5y agoI dunno about all that, but in my case what I mean is "the only choice that would satisfy the author's criteria of being statistically sound"
- xpe 5y agoYes, in some cases, teams want an experiment to back them up. If a decision maker cannot dedicate the necessary resources to properly power a test, it arguably would be more rational to use another process to make the decision. Perhaps a thoughtfully designed team discussion or even a 'Person P will make the decision because the uncertainty is relatively high, consensus is relatively low, and additional discussion is unlikely to clarify -- meanwhile we could go ahead and try the thing and see what happens!"
- tqi 5y ago> If a decision maker cannot dedicate the necessary resources to properly power a test Apologies if this is what you meant, but in a lot of cases it's not a matter of not dedicating enough resources so much as there aren't enough resources (ie visitors, users, posts, etc) to detect reasonable effect size. And in those cases, it seems like the author's point is that running a test is actively worse than a gut call or focus group or whatever else. But to your point, product teams often don't like gut calls because they feel subjective and unilateral, so having a test allows some degree of shared accountability for decisions.
- deleted 5y ago[deleted]
- xpe 5y ago> Significance is the probability of seeing an effect where no effect exists. In my experience, the term significance level is more common and clearer. From Wikipedia: > More precisely, a study's defined significance level, denoted by α (alpha), is the probability of the study rejecting the null hypothesis, given that the null hypothesis is true; and the p-value of a result, p, is the probability of obtaining a result at least as extreme, given that the null hypothesis is true. The result is statistically significant, by the standards of the study, when p ≤ α.
- xpe 5y agoAlso, I prefer to use the term statistical significance over just significance because it is clearer to a broad audience across fields. From Wikipedia: > The term significance does not imply importance here, and the term statistical significance is not the same as research significance, theoretical significance, or practical significance. For example, the term clinical significance refers to the practical importance of a treatment effect.
- PheonixPharts 5y agoTo preface this comment, I generally really like Evan Miller's writing on statistics, but this statement: > anyone who lacks a firm understanding of statistical power should not be designing or interpreting A/B tests. Completely ignores that you can approach A/B testing from an entirely Bayesian perspective (which Miller has written about fairly well in the past). I might be a bit biased in favor of Bayesian methods, but I'm honestly a bit surprised that many people still run A/B tests using a frequentist frameworks such as this. The biggest reason I would argue to dismiss the NHST (Null-hypothesis significance test) approaching is because A/B Tests are not really controlled experiments, at least not in the same way clinical trials are. User behavior is always observational even if you have a control group. Whether you go NHST or Bayesian is a coin toss in a controlled environment, but when looking at user behavior on a web site it's much better to think in Bayesian terms. Other reasons why a Bayesian frame work should be used: - Marketers care about "Probability A is better than B", not "failure to reject null hypothesis". - If you've been running A/B tests for years, you most certainly have a very good prior over the distribution of conversion rates. It's literally wasting your time and money not to make use of these. - If you're measuring human behavior on a website, there are absolutely confounding variables at play and you want to model these explicitly. It's much easier to do this when thinking in a Bayesian framework. I've been designing and running A/B test of all sorts of complexity at all sorts of companies for a very long time, and my experience has been that approaching the problem from a Bayesian perspective is by far the superior way to run your tests.
- ehassler 5y agoMany A/B tests as RCTs do suffer from the fact that they're randomizations of convenience sample of users coming to a web site, but it's always seemed to me that randomization addresses most confounders due to population heterogeneity. Is there some specific confounding variables you worry about that are not addressed by randomization in this way? One challenge for me is how to explain to marketers what the "Probability A is better than B" means from a Bayesian perspective. How do you convey to them that such a probability is not a frequency of an event? I haven't met anyone who is not a statistician that won't default to "you know, how often it happens" when pressed to define probability. My attempts to engage marketers on this fall like water off a duck's back. In general I prefer NHST methods because I often require reasonably tight error control. Do you do any kind of calibration to ensure error rate control when doing your procedures? Or are your clients comfortable with less tight inferential error controls?
- tunesmith 5y agoWhat's the difference between power and sensitivity?
- j7ake 5y agoCan these problems be circumvented by thinking of non parametric bootstrap methods to directly infer significance from an empirical distribution of not-significant results?