10 ms·

p-Hacking and False Discovery in A/B Testing

299 points by gwern 8y ago

nvahalik 8y ago
ELI5... what’s a p-hack?
- laurieg 8y ago
  Doing the same experiment again and again until you get the answer you want.
  - drdrey 8y ago
    In this case though I think they mean stopping the experiment early if there seems to be a positive result rather than experimenting again
- SquareWheel 8y ago
  Not an easy ELI5, but consider these resources. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging https://www.youtube.com/watch?v=bf3egy7TQ2Q https://www.youtube.com/watch?v=bf3egy7TQ2Q
- danielvf 8y ago
  “the misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect. This is done by performing many statistical tests on the data and only paying attention to those that come back with significant results, instead of stating a single hypothesis about an underlying effect before the analysis and then conducting a single test for it.“ https://en.m.wikipedia.org/wiki/Data_dredging https://en.m.wikipedia.org/wiki/Data_dredging And here’s a great example of real life P-Hacking to get a catchy article about the health benefits of chocolate: https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
  - vitus 8y ago
    FiveThirtyEight actually has a pretty good demo of p-hacking that demonstrates how one underlying dataset can be used to derive any desired conclusion(s) by deciding which factors to include / exclude. https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/
  - cryptonector 8y ago
    Torturing the data until it sings.
- delecti 8y ago
  In statistics, a p-value [1] is a the odds of getting a result by chance if there's not actually an effect from the thing you're testing. A common threshold to use is .05, which means that you'd expect the given result less than 5% of the time by chance. Typically this indicates a relationship, but if you simultaneously test enough things that aren't causally linked you'd expect some to have p<0.05. A good example is this xkcd [2]. In the example, green jelly beans aren't actually related to acne, but just by chance the result seems to indicate they are. [1] https://en.m.wikipedia.org/wiki/P-value https://en.m.wikipedia.org/wiki/P-value [2] https://xkcd.com/882/ https://xkcd.com/882/
- patmcc 8y ago
  https://xkcd.com/882/ https://xkcd.com/882/
  - vecter 8y ago
    This is not p-hacking.
    - soberhoff 8y ago
      It is.
- cepth 8y ago
  If you’ve taken some statistics or econometrics, you’ve probably heard of “significance levels” and “p-values”. For some reason, academia choose 0.05 as a threshold for “meaningful” or “significant” results. Generally, a 0.05 p-value means that you would observe your result in 5% of experiments due to random sampling error. I.e. if I tested “is X correlated with cancer”, and my null hypothesis is “X isn’t correlated with cancer”, a 0.05 p-value would meet the threshold to reject that null hypothesis. Generally, a lower p-value means a more statistically significant result. The problem is that 0.05 seems to be much too high of a p-value. I.e. clever experimental design and cherry picking can generate many results that are statistically significant at that level. Many academics advocate for moving to a 0.01 or even 0.001 significance threshold. Recently, in some academic fields, there’s been widespread concern that many research studies were p-hacked. See for example, this paper that blew up last year in the finance community, because it suggests a significant number of finance papers, including some seminal ones, had p-hacked results: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677. The counter-argument is that for certain scientific fields, you may never be able to reach a p-value threshold of 0.001. This means the vast majority of research couldn’t be published in journals, academics wouldn’t be able to get promoted etc.
  - hammock 8y ago
    This is wrong. P hacking has nothing to do with the p being too lenient.
    - Fomite 8y ago
      This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.
      - cepth 8y ago
        > This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to address the lack of reproducibility is not something that I made up. Yes, lowering the p-value threshold does not eliminate the motivations/techniques that are necessary for p-hacking, but it can make it a lot harder, and a lot less worthwhile. If you work in academia, and it takes you much longer to now cherrypick a sample to meet a much lower p-cutoff, it seems to follow that we would see less of it. This is an excerpt from a paper soon to be published in Nature: https://imai.princeton.edu/research/files/significance.pdf https://imai.princeton.edu/research/files/significance.pdf. The key quote: 'We have diverse views about how best to improve reproducibility,and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.' With regards to the comment that you could p-hack up to any strictness, I'm not sure this is correct. If you accept the proposal laid out in that Nature paper, to lower the threshold to P<0.005, or if we go even lower to P<0.001 I don't believe that you'd be able to p-hack in any practical way. Yes, you could cherry pick a tiny sample, but any peer reviewer or colleague of yours is going to ask questions about the sample.
- azernik 8y ago
  Even better than examples and explanations, learn by doing: https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/ "You’re a social scientist with a hunch: The U.S. economy is affected by whether Republicans or Democrats are in office. Try to show that a connection exists, using real data going back to 1948. For your results to be publishable in an academic journal, you’ll need to prove that they are “statistically significant” by achieving a low enough p-value."
raphaelrk 8y ago
Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?
- t3scrote 8y ago
  We set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers into a bayesian calculator, and call it a winner of there is a 90%+ probability that the treatment beats the control. https://www.abtestguide.com/bayesian/ https://www.abtestguide.com/bayesian/
  - tzahola 8y ago
    So one in ten of your findings is bogus.
    - t3scrote 8y ago
      But isn’t that better than blindly introducing changes without testing them at all?
  - joshuamorton 8y ago
    Why so high? Beysian results aren't p-values, a 60-70% probably that treatment beats control is just that, not a pvalue of .4 or .3 (which would say nothing).
babl-yc 8y ago
How can A/B testing tools be improved to prevent p-value hacking? Could it be as simple as declaring your test duration before starting the experiment, and having the tool add an asterisk to your results if you stop the experiment early?
- stenl 8y ago
  They should use Bayesian statistics, in which case it doesn’t matter when you stop (more precisely, stopping when you get a result does not bias the outcome; of course running the test longer will make the result more robust). See http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-analysis/ http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a...
- hackandtrip 8y ago
  - You must choose the sample size BEFORE - You must choose the significance level BEFORE (0.05 is not always the best choice)
- regularfry 8y ago
  Optimizely claim to have since fixed this. The paper where they introduce the method is here: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
aisscott 8y ago
Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper. If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://www.optimizely.com/resources/stats-engine-whitepaper/ https://www.optimizely.com/resources/stats-engine-whitepaper...
- boron1006 8y ago
  I'm curious about the wording "effects are truly null". I was always under the impression that you can never really "accept the null", but rather "fail to reject the alternative".
  - windows_tips 8y ago
    The "null" hypothesis would typically be that "there is no effect". All that you can really do is prove it wrong, by measuring an effect when there "should", by the hypothesis, be none. Due to what is known as the "problem of induction", it's not sufficient to accept a hypothesis because you appeared to not measure an effect in the past, as that says nothing about whether an effect will occur the next time a measurement is made. p-value is the "chance" of measuring an effect, given that no effect actually occurred.
  - gwern 8y ago
    In your standard NHST test, sure. But you can do different models. If I'm reading OP right, what they do is a mixture model, in which effects are assumed to come either from a zero-mean distribution or a positive-mean distribution with unknown probability _P_/1-_P_ and then you fit the collection of >2k effect sizes to find out what value of _P_ best fits the dataset as a whole. Apparently the best fit assumes that ~70% of effects are actually ~0. This can also be done nicely with a Bayesian mixture model or a spike-and-slab multilevel model, and that is what is done in "What works in e-commerce - a meta-analysis of 6700 online experiments", Brown & Jones 2017 http://www.qubit.com/sites/default/files/pdf/qubit_meta_analysis.pdf http://www.qubit.com/sites/default/files/pdf/qubit_meta_anal... (although they don't formulate it in terms of a sharp null but ask the more relevant 'probability of a >0 beneficial effect', which for some kinds of A/B test has a very low prior - like only 15% for 'back to top' A/B tests).
cle 8y ago
Traditional A/B testing has very poor ergonomics. Experimenters are usually put in awkward conflict-of-interest situations that create multiple strong incentives not to perform rigorous, disciplined, valid experiments. Null hypothesis significance testing is fundamentally misaligned with business needs and is not a good tool for businesses. This is true in many fields of science as well, but at least they have some mechanisms that try to ensure that experiments are unbiased. Businesses often don't have the same internal and external incentives that lead to those mechanisms, and so NHST is abused even more.
- havkom 8y ago
  Well, I have had well run experiments which showed that the “currently internally hyped” way of doing things is completely inferior to the “old boring” and inexpensive way. The project leader was extremely hyped about doing the experiment to prove the superiority of the new hyped way. When he got the results though, it was clear that this was not to be talked about and this result would not be presented to his superiors.
  - tzahola 8y ago
    “We’re a data-driven organization! [as long as the data fits our agenda]” It’s one of my favorite methodologies, next to “agile waterfall” and “holocracy with managers, middle-managers and minibosses”.
boron1006 8y ago
I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and run experiments for predetermined amounts of time. I don't think the tech environment is very conducive to running experiments. Everything moves too fast, by the time you figure out the results someone gives you are bs, they've already got promoted 3 times and work as a director at a different company. I work in science now, and although people still p-hack like hell, there's at least some sort of shame about it. There's a long term cost too, I've met a couple researchers who have spent years trying to replicate some finding they got early in their career through suspicious means.
- pimmen 8y ago
  I've seen that culture too. I don't want to throw my colleagues under the bus but the other teams at my company that have done A/B testing to compare design choices don't perform proper controls or pre-register hypothesis. Eg. when they want to divide the two groups, they often divide them based on some classification (gender, age, geography etc.) rather than dividing them up completely randomly. In my team we try to be methodical. I'm just a lowly engineer but one of my team mates is a statistician and another is a PhD student. We know we need to pre-register the hypothesis and at what significance we're testing for first, divide the groups randomly and run the experiment for a set time period. We've gotten a lot more negative results and been proven wrong in our guesses than the other teams. For some reason I take pride in that.
- tedsanders 8y ago
  Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to treat 1,000 patients, thinking that the effect would be small. This is one reason I think Bayesian approaches are better than frequentist approaches for A/B testing.
nanis 8y ago
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a thing.
- p10_user 8y ago
  It’s done in academia too. Along with “remove the outlier data that messes up our p values”
  - glup 8y ago
    The difference is that the people in academia know that it is statistically unsound and choose to act unethically. I think the problem in A/B testing is that a lot of developers don't know it is unsound.
    - carlmr 8y ago
      This is one of the cases where self-taught developers are usually not as good as those with a "proper" education. In uni you'll learn a lot of tangentially related stuff like p-hacking and design of experiment which a lot of people won't pick up when self-taught.
  - whyever 8y ago
    It might make sense to remove systematic outliers if you know they are from non-statistical effects.
- IshKebab 8y ago
  It's perfectly fine to run an experiment until you get a significant result. You just have to do the maths differently - that's what most people don't know.
  - 8y ago
dahdum 8y ago
After listening to Optimizely reps give a talk about their success with a client (who was present) I suspect the support reps encourage these false positives. They presented a few tests as fantastic wins, when they all had basic flaws (like cold audience vs self selected for interest). Maybe that was just one bad apple (doubtful)...but it was a large client and someone they felt should represent the company as a speaker. Concerns from the audience were dismissed and referred to follow up after the talk. Never thought the same of Optimizely after that.
emodendroket 8y ago
Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology
- Engineering-MD 8y ago
  I would say it is. By not being hypothesis driven, you are producing too many degrees of freedom by de facto testing every comparison. This is a huge problem in science too. I have regularly been told to just see what happens and come up with hypotheses after, or others have been unable to say what their hypotheses actually are. Scientists seem less and less likely to be trained in statistics, and in the scientific process. Technical knowledge is important, but understanding of the scientific process is much more important.
  - emodendroket 8y ago
    There was a scandal because it turned out that was essentially what Brian Wansink was up to with his studies of food; that's kind of what put me on to this line of thinking.
- andreareina 8y ago
  I think that unnecessarily discounts exploratory work. There's nothing wrong with forming a hypothesis after seeing a pattern in data. But remember that it's just a hypothesis -- an unconfirmed guess. After the hypothesis is formulated, then an experiment can be designed to test it, see if its predictions hold up.
  - deleted 8y ago
    [deleted]
  - emodendroket 8y ago
    I don't know about you, but the "exploratory" work is, in my experience, the start and end of it.
User23 8y ago
I had the enjoyable experience of sitting at a tech conference and listening to the others in my group tell one of my friends that he had no idea what he was talking about when he said they weren't designing a proper experiment. I was the only one there who knew he's a particle physicist. The OP is horrifyingly right.
paraschopra 8y ago
Hi, founder of VWO here. We revamped our testing engine to Bayesian in 2015 to prevent the ‘peeking problem’ with frequentists approaches. You can read about our approach https://vwo.com/blog/smartstats-testing-for-truth/ https://vwo.com/blog/smartstats-testing-for-truth/
- paraschopra 8y ago
  Here’s the math of our Bayesian testing engine (for those who are interested in knowing how we do it) https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic...
- clircle 8y ago
  Sequential techniques like Wald's SPRT don't have this problem.
raverbashing 8y ago
People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't Assuming 'B' is the new option, there are 3 possibilities, A is better than B, A is equivalent to B, A is worse than B If your p-hacked experiment tells you to change from A to B while the null hypothesis was correct, you didn't get much worse off than you were in the first place. And if your long term metrics were in place then you can get a better measure for your experiment. Not to mention experimental failures by unaccounted variables
- danieltillett 8y ago
  Yes not to mention that for nearly all A/B tests the size of the effect is minimal anyway. I have found that if the effect is large you don't need statistics and if it is small it doesn't matter.
- dahdum 8y ago
  A large percentage of experiments I've run were intended to only test if variant was worse than control, we didn't care much how much better the variant may be. Usually these would be positive consumer facing features we were concerned may negatively affect conversion. The switch to Bayesian made that a lot easier to run.
gingerlime 8y ago
I created an open source A/B test framework[0], which also uses Bayesian analysis on the dashboard. IANAS(tatistician), but from what I understand it’s still better to plan the check point in advance, rather than stop when reaching significance. A couple of articles worth reading [1] [2] (can’t exactly vouch for their validity but seem to make some good arguments that appear thought out) [0] https://github.com/Alephbet/gimel https://github.com/Alephbet/gimel [1] http://varianceexplained.org/r/bayesian-ab-testing/ http://varianceexplained.org/r/bayesian-ab-testing/ [2] http://blog.analytics-toolkit.com/2017/the-bane-of-ab-testing-reaching-statistical-significance/ http://blog.analytics-toolkit.com/2017/the-bane-of-ab-testin...
baybal2 8y ago
Seconding this, the number of e-commerce companies who got "A/B testing into bankruptcy" on my memory approaches 20. My take on this. Even in cases where such testing was done by disciplined statisticians (which is not the case in at least 9 times out of 10. Yes, a math or cs PhD major is not a professional statistician by any stretch,) the value of advice made from that data is marginal at best. As eCommerce is bread and butter of cheap electronics industry, I saw times and times again that "science driven" outfits loose out to others. Not so much because of their quality of decision making was demonstrably inferior, but because their obsession with "statistical tasseography" drained their resources, and shifted their focus away from things of obvious importance.
yichijin 8y ago
Hi all. Jimmy, statistician from Optimizely chiming in. We were excited to collaborate with the authors on this study. Keep in mind the data used in this analysis is from 2014, before we introduced sequential testing and FDR correction as to specifically address this p-hacking issue. I expect these results are in line with any platform using fixed-horizon frequentist methods. Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...
geoprofi 8y ago
My very recent meta-analysis of 115 A/B tests reveals that a large proportion are highly suspect for p-hacking: http://blog.analytics-toolkit.com/2018/analysis-of-115-a-b-tests-average-lift-statistical-power/ http://blog.analytics-toolkit.com/2018/analysis-of-115-a-b-t... Going the Bayesian way, as suggested in some comments, is no solution at all, as I am not aware of an accepted Bayesian approach to dealing with the issue: http://blog.analytics-toolkit.com/2017/bayesian-ab-testing-not-immune-to-optional-stopping-issues/ http://blog.analytics-toolkit.com/2017/bayesian-ab-testing-n... (feel free to run sims, if you do not trust the logic ;-)) as well as on a more general level: http://blog.analytics-toolkit.com/2017/5-reasons-bayesian-ab-testing-debunked/ http://blog.analytics-toolkit.com/2017/5-reasons-bayesian-ab...

299 points by gwern 8y ago

nvahalik 8y ago

ELI5... what’s a p-hack?

laurieg 8y ago
Doing the same experiment again and again until you get the answer you want.
- drdrey 8y ago
  In this case though I think they mean stopping the experiment early if there seems to be a positive result rather than experimenting again
SquareWheel 8y ago
Not an easy ELI5, but consider these resources. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging https://www.youtube.com/watch?v=bf3egy7TQ2Q https://www.youtube.com/watch?v=bf3egy7TQ2Q
danielvf 8y ago
“the misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect. This is done by performing many statistical tests on the data and only paying attention to those that come back with significant results, instead of stating a single hypothesis about an underlying effect before the analysis and then conducting a single test for it.“ https://en.m.wikipedia.org/wiki/Data_dredging https://en.m.wikipedia.org/wiki/Data_dredging And here’s a great example of real life P-Hacking to get a catchy article about the health benefits of chocolate: https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
- vitus 8y ago
  FiveThirtyEight actually has a pretty good demo of p-hacking that demonstrates how one underlying dataset can be used to derive any desired conclusion(s) by deciding which factors to include / exclude. https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/
- cryptonector 8y ago
  Torturing the data until it sings.
delecti 8y ago
In statistics, a p-value [1] is a the odds of getting a result by chance if there's not actually an effect from the thing you're testing. A common threshold to use is .05, which means that you'd expect the given result less than 5% of the time by chance. Typically this indicates a relationship, but if you simultaneously test enough things that aren't causally linked you'd expect some to have p<0.05. A good example is this xkcd [2]. In the example, green jelly beans aren't actually related to acne, but just by chance the result seems to indicate they are. [1] https://en.m.wikipedia.org/wiki/P-value https://en.m.wikipedia.org/wiki/P-value [2] https://xkcd.com/882/ https://xkcd.com/882/
patmcc 8y ago
https://xkcd.com/882/ https://xkcd.com/882/
- vecter 8y ago
  This is not p-hacking.
  - soberhoff 8y ago
    It is.
cepth 8y ago
If you’ve taken some statistics or econometrics, you’ve probably heard of “significance levels” and “p-values”. For some reason, academia choose 0.05 as a threshold for “meaningful” or “significant” results. Generally, a 0.05 p-value means that you would observe your result in 5% of experiments due to random sampling error. I.e. if I tested “is X correlated with cancer”, and my null hypothesis is “X isn’t correlated with cancer”, a 0.05 p-value would meet the threshold to reject that null hypothesis. Generally, a lower p-value means a more statistically significant result. The problem is that 0.05 seems to be much too high of a p-value. I.e. clever experimental design and cherry picking can generate many results that are statistically significant at that level. Many academics advocate for moving to a 0.01 or even 0.001 significance threshold. Recently, in some academic fields, there’s been widespread concern that many research studies were p-hacked. See for example, this paper that blew up last year in the finance community, because it suggests a significant number of finance papers, including some seminal ones, had p-hacked results: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3017677. The counter-argument is that for certain scientific fields, you may never be able to reach a p-value threshold of 0.001. This means the vast majority of research couldn’t be published in journals, academics wouldn’t be able to get promoted etc.
- hammock 8y ago
  This is wrong. P hacking has nothing to do with the p being too lenient.
  - Fomite 8y ago
    This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time.
    - cepth 8y ago
      > This is wrong. P hacking has nothing to do with the p being too lenient. > This. I can p-hack in my field, if I wanted to, up to a p-value of arbitrary strictness, given enough time. I'm not a practicing scientist/academic, so I want to be careful here. But, I think both of you are being a little uncharitable/pedantic. P-hacking is one contributor to the broader reproducibility crisis. Lowering the p-value to address the lack of reproducibility is not something that I made up. Yes, lowering the p-value threshold does not eliminate the motivations/techniques that are necessary for p-hacking, but it can make it a lot harder, and a lot less worthwhile. If you work in academia, and it takes you much longer to now cherrypick a sample to meet a much lower p-cutoff, it seems to follow that we would see less of it. This is an excerpt from a paper soon to be published in Nature: https://imai.princeton.edu/research/files/significance.pdf https://imai.princeton.edu/research/files/significance.pdf. The key quote: 'We have diverse views about how best to improve reproducibility,and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.' With regards to the comment that you could p-hack up to any strictness, I'm not sure this is correct. If you accept the proposal laid out in that Nature paper, to lower the threshold to P<0.005, or if we go even lower to P<0.001 I don't believe that you'd be able to p-hack in any practical way. Yes, you could cherry pick a tiny sample, but any peer reviewer or colleague of yours is going to ask questions about the sample.
azernik 8y ago
Even better than examples and explanations, learn by doing: https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/ "You’re a social scientist with a hunch: The U.S. economy is affected by whether Republicans or Democrats are in office. Try to show that a connection exists, using real data going back to 1948. For your results to be publishable in an academic journal, you’ll need to prove that they are “statistically significant” by achieving a low enough p-value."

raphaelrk 8y ago

Optimizely being for large enterprises, curious how people do A/B tests at their respective startups. Do most roll their own? How do you make sure your science is sound?

t3scrote 8y ago
We set audience criteria where the user account must be created after the test launches, from there its a 50/50 split control/treatment experience (based on user id). The metric we are optimizing for is almost always conversion rate. We will turn the experiment off early if the treatment group is having really poor numbers, otherwise once about 4000 accounts have been entered into the experiment we plug the numbers into a bayesian calculator, and call it a winner of there is a 90%+ probability that the treatment beats the control. https://www.abtestguide.com/bayesian/ https://www.abtestguide.com/bayesian/
- tzahola 8y ago
  So one in ten of your findings is bogus.
  - t3scrote 8y ago
    But isn’t that better than blindly introducing changes without testing them at all?
- joshuamorton 8y ago
  Why so high? Beysian results aren't p-values, a 60-70% probably that treatment beats control is just that, not a pvalue of .4 or .3 (which would say nothing).

babl-yc 8y ago

How can A/B testing tools be improved to prevent p-value hacking? Could it be as simple as declaring your test duration before starting the experiment, and having the tool add an asterisk to your results if you stop the experiment early?

stenl 8y ago
They should use Bayesian statistics, in which case it doesn’t matter when you stop (more precisely, stopping when you get a result does not bias the outcome; of course running the test longer will make the result more robust). See http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-analysis/ http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a...
hackandtrip 8y ago
- You must choose the sample size BEFORE - You must choose the significance level BEFORE (0.05 is not always the best choice)
regularfry 8y ago
Optimizely claim to have since fixed this. The paper where they introduce the method is here: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...

aisscott 8y ago

Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper. If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://www.optimizely.com/resources/stats-engine-whitepaper/ https://www.optimizely.com/resources/stats-engine-whitepaper...

boron1006 8y ago
I'm curious about the wording "effects are truly null". I was always under the impression that you can never really "accept the null", but rather "fail to reject the alternative".
- windows_tips 8y ago
  The "null" hypothesis would typically be that "there is no effect". All that you can really do is prove it wrong, by measuring an effect when there "should", by the hypothesis, be none. Due to what is known as the "problem of induction", it's not sufficient to accept a hypothesis because you appeared to not measure an effect in the past, as that says nothing about whether an effect will occur the next time a measurement is made. p-value is the "chance" of measuring an effect, given that no effect actually occurred.
- gwern 8y ago
  In your standard NHST test, sure. But you can do different models. If I'm reading OP right, what they do is a mixture model, in which effects are assumed to come either from a zero-mean distribution or a positive-mean distribution with unknown probability _P_/1-_P_ and then you fit the collection of >2k effect sizes to find out what value of _P_ best fits the dataset as a whole. Apparently the best fit assumes that ~70% of effects are actually ~0. This can also be done nicely with a Bayesian mixture model or a spike-and-slab multilevel model, and that is what is done in "What works in e-commerce - a meta-analysis of 6700 online experiments", Brown & Jones 2017 http://www.qubit.com/sites/default/files/pdf/qubit_meta_analysis.pdf http://www.qubit.com/sites/default/files/pdf/qubit_meta_anal... (although they don't formulate it in terms of a sharp null but ask the more relevant 'probability of a >0 beneficial effect', which for some kinds of A/B test has a very low prior - like only 15% for 'back to top' A/B tests).

cle 8y ago

Traditional A/B testing has very poor ergonomics. Experimenters are usually put in awkward conflict-of-interest situations that create multiple strong incentives not to perform rigorous, disciplined, valid experiments. Null hypothesis significance testing is fundamentally misaligned with business needs and is not a good tool for businesses. This is true in many fields of science as well, but at least they have some mechanisms that try to ensure that experiments are unbiased. Businesses often don't have the same internal and external incentives that lead to those mechanisms, and so NHST is abused even more.

havkom 8y ago
Well, I have had well run experiments which showed that the “currently internally hyped” way of doing things is completely inferior to the “old boring” and inexpensive way. The project leader was extremely hyped about doing the experiment to prove the superiority of the new hyped way. When he got the results though, it was clear that this was not to be talked about and this result would not be presented to his superiors.
- tzahola 8y ago
  “We’re a data-driven organization! [as long as the data fits our agenda]” It’s one of my favorite methodologies, next to “agile waterfall” and “holocracy with managers, middle-managers and minibosses”.

boron1006 8y ago

I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and run experiments for predetermined amounts of time. I don't think the tech environment is very conducive to running experiments. Everything moves too fast, by the time you figure out the results someone gives you are bs, they've already got promoted 3 times and work as a director at a different company. I work in science now, and although people still p-hack like hell, there's at least some sort of shame about it. There's a long term cost too, I've met a couple researchers who have spent years trying to replicate some finding they got early in their career through suspicious means.

pimmen 8y ago
I've seen that culture too. I don't want to throw my colleagues under the bus but the other teams at my company that have done A/B testing to compare design choices don't perform proper controls or pre-register hypothesis. Eg. when they want to divide the two groups, they often divide them based on some classification (gender, age, geography etc.) rather than dividing them up completely randomly. In my team we try to be methodical. I'm just a lowly engineer but one of my team mates is a statistician and another is a PhD student. We know we need to pre-register the hypothesis and at what significance we're testing for first, divide the groups randomly and run the experiment for a set time period. We've gotten a lot more negative results and been proven wrong in our guesses than the other teams. For some reason I take pride in that.
tedsanders 8y ago
Running experiments for preset lengths is a mistake. If an effect is strong, it will show up earlier. If this is the case, you want to be able to switch earlier. If you are running a drug trial of drug A vs a control, and drug A kills 100% of the first 100 patients who take it while the control kills 0, you end the trial immediately. You don’t continue to give it to 900 patients just because you pre-registered to treat 1,000 patients, thinking that the effect would be small. This is one reason I think Bayesian approaches are better than frequentist approaches for A/B testing.

nanis 8y ago

Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a thing.

p10_user 8y ago
It’s done in academia too. Along with “remove the outlier data that messes up our p values”
- glup 8y ago
  The difference is that the people in academia know that it is statistically unsound and choose to act unethically. I think the problem in A/B testing is that a lot of developers don't know it is unsound.
  - carlmr 8y ago
    This is one of the cases where self-taught developers are usually not as good as those with a "proper" education. In uni you'll learn a lot of tangentially related stuff like p-hacking and design of experiment which a lot of people won't pick up when self-taught.
- whyever 8y ago
  It might make sense to remove systematic outliers if you know they are from non-statistical effects.
IshKebab 8y ago
It's perfectly fine to run an experiment until you get a significant result. You just have to do the maths differently - that's what most people don't know.
- 8y ago

dahdum 8y ago

After listening to Optimizely reps give a talk about their success with a client (who was present) I suspect the support reps encourage these false positives. They presented a few tests as fantastic wins, when they all had basic flaws (like cold audience vs self selected for interest). Maybe that was just one bad apple (doubtful)...but it was a large client and someone they felt should represent the company as a speaker. Concerns from the audience were dismissed and referred to follow up after the talk. Never thought the same of Optimizely after that.

emodendroket 8y ago

Isn't randomly looking for a pattern and then slapping a hypothesis on it post facto a form of "p-hacking"? Because that's completely commonplace and unremarkable practice in technology

Engineering-MD 8y ago
I would say it is. By not being hypothesis driven, you are producing too many degrees of freedom by de facto testing every comparison. This is a huge problem in science too. I have regularly been told to just see what happens and come up with hypotheses after, or others have been unable to say what their hypotheses actually are. Scientists seem less and less likely to be trained in statistics, and in the scientific process. Technical knowledge is important, but understanding of the scientific process is much more important.
- emodendroket 8y ago
  There was a scandal because it turned out that was essentially what Brian Wansink was up to with his studies of food; that's kind of what put me on to this line of thinking.
andreareina 8y ago
I think that unnecessarily discounts exploratory work. There's nothing wrong with forming a hypothesis after seeing a pattern in data. But remember that it's just a hypothesis -- an unconfirmed guess. After the hypothesis is formulated, then an experiment can be designed to test it, see if its predictions hold up.
- deleted 8y ago
  [deleted]
- emodendroket 8y ago
  I don't know about you, but the "exploratory" work is, in my experience, the start and end of it.

User23 8y ago

I had the enjoyable experience of sitting at a tech conference and listening to the others in my group tell one of my friends that he had no idea what he was talking about when he said they weren't designing a proper experiment. I was the only one there who knew he's a particle physicist. The OP is horrifyingly right.

paraschopra 8y ago

Hi, founder of VWO here. We revamped our testing engine to Bayesian in 2015 to prevent the ‘peeking problem’ with frequentists approaches. You can read about our approach https://vwo.com/blog/smartstats-testing-for-truth/ https://vwo.com/blog/smartstats-testing-for-truth/

paraschopra 8y ago
Here’s the math of our Bayesian testing engine (for those who are interested in knowing how we do it) https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic...
clircle 8y ago
Sequential techniques like Wald's SPRT don't have this problem.

raverbashing 8y ago

People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't Assuming 'B' is the new option, there are 3 possibilities, A is better than B, A is equivalent to B, A is worse than B If your p-hacked experiment tells you to change from A to B while the null hypothesis was correct, you didn't get much worse off than you were in the first place. And if your long term metrics were in place then you can get a better measure for your experiment. Not to mention experimental failures by unaccounted variables

danieltillett 8y ago
Yes not to mention that for nearly all A/B tests the size of the effect is minimal anyway. I have found that if the effect is large you don't need statistics and if it is small it doesn't matter.
dahdum 8y ago
A large percentage of experiments I've run were intended to only test if variant was worse than control, we didn't care much how much better the variant may be. Usually these would be positive consumer facing features we were concerned may negatively affect conversion. The switch to Bayesian made that a lot easier to run.

gingerlime 8y ago

I created an open source A/B test framework[0], which also uses Bayesian analysis on the dashboard. IANAS(tatistician), but from what I understand it’s still better to plan the check point in advance, rather than stop when reaching significance. A couple of articles worth reading [1] [2] (can’t exactly vouch for their validity but seem to make some good arguments that appear thought out) [0] https://github.com/Alephbet/gimel https://github.com/Alephbet/gimel [1] http://varianceexplained.org/r/bayesian-ab-testing/ http://varianceexplained.org/r/bayesian-ab-testing/ [2] http://blog.analytics-toolkit.com/2017/the-bane-of-ab-testing-reaching-statistical-significance/ http://blog.analytics-toolkit.com/2017/the-bane-of-ab-testin...

baybal2 8y ago

Seconding this, the number of e-commerce companies who got "A/B testing into bankruptcy" on my memory approaches 20. My take on this. Even in cases where such testing was done by disciplined statisticians (which is not the case in at least 9 times out of 10. Yes, a math or cs PhD major is not a professional statistician by any stretch,) the value of advice made from that data is marginal at best. As eCommerce is bread and butter of cheap electronics industry, I saw times and times again that "science driven" outfits loose out to others. Not so much because of their quality of decision making was demonstrably inferior, but because their obsession with "statistical tasseography" drained their resources, and shifted their focus away from things of obvious importance.

yichijin 8y ago

Hi all. Jimmy, statistician from Optimizely chiming in. We were excited to collaborate with the authors on this study. Keep in mind the data used in this analysis is from 2014, before we introduced sequential testing and FDR correction as to specifically address this p-hacking issue. I expect these results are in line with any platform using fixed-horizon frequentist methods. Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...

geoprofi 8y ago

My very recent meta-analysis of 115 A/B tests reveals that a large proportion are highly suspect for p-hacking: http://blog.analytics-toolkit.com/2018/analysis-of-115-a-b-tests-average-lift-statistical-power/ http://blog.analytics-toolkit.com/2018/analysis-of-115-a-b-t... Going the Bayesian way, as suggested in some comments, is no solution at all, as I am not aware of an accepted Bayesian approach to dealing with the issue: http://blog.analytics-toolkit.com/2017/bayesian-ab-testing-not-immune-to-optional-stopping-issues/ http://blog.analytics-toolkit.com/2017/bayesian-ab-testing-n... (feel free to run sims, if you do not trust the logic ;-)) as well as on a more general level: http://blog.analytics-toolkit.com/2017/5-reasons-bayesian-ab-testing-debunked/ http://blog.analytics-toolkit.com/2017/5-reasons-bayesian-ab...