4 ms·

The p-value represents "Probability this is all luck". So why isn't 1 - p "Probability this isn't all luck"? >> Another extremely important point is that the c

by meric 15y ago

The p-value represents "Probability this is all luck". So why isn't 1 - p "Probability this isn't all luck"? >> Another extremely important point is that the classical >> statistics you learned to use to decide that something >> was <5% likely to arise by chance, only apply if you >> decided in advance to do exactly that many trials and >> then stop. I agree. Not doing that I'd say it's "fudging with the numbers". I can't find on their article where they did this though.

jharsman 15y ago
A better interpretation might be "probability this would happen if results were governed purely by chance". Note that the distinction is important if the process is in fact governed by chance! Lets say I roll a dice two times and get a six both times. The probability of this happening is 1/36 or about 3%. Would you say I have established with a 95% confidence interval that the particular die I'm using always rolls six? No, because you have good reason to believe that the the results are in fact random. Or in other words, you have a strong prior belief that the hypothesis you're testing is false.
- meric 15y ago
  >> Or in other words, you have a strong prior belief that the hypothesis you're testing is false. Yes, that is true for the dice roll; You already know most die aren't rigged. I don't see how this affects the results of optimizely's product; You do not have a strong prior belief if the hypothesis is true or false. Also note that the number of observations in the article used were in the thousands.
jules 15y ago
The p-value means "chance of this happening, if the two were equally effective". So 1-p means "change of this NOT happening, if the two were equally effective". Note that in frequentist statistics, the sentence "Probability this is all luck" doesn't even make sense. Either the universe is so that it was luck, or it is not. In frequentist statistics you do not assign probabilities to models of the universe. You only assign probabilities to observations, given a fixed model of the universe.

by meric 15y ago

jharsman 15y ago

A better interpretation might be "probability this would happen if results were governed purely by chance". Note that the distinction is important if the process is in fact governed by chance! Lets say I roll a dice two times and get a six both times. The probability of this happening is 1/36 or about 3%. Would you say I have established with a 95% confidence interval that the particular die I'm using always rolls six? No, because you have good reason to believe that the the results are in fact random. Or in other words, you have a strong prior belief that the hypothesis you're testing is false.

meric 15y ago
>> Or in other words, you have a strong prior belief that the hypothesis you're testing is false. Yes, that is true for the dice roll; You already know most die aren't rigged. I don't see how this affects the results of optimizely's product; You do not have a strong prior belief if the hypothesis is true or false. Also note that the number of observations in the article used were in the thousands.

jules 15y ago

The p-value means "chance of this happening, if the two were equally effective". So 1-p means "change of this NOT happening, if the two were equally effective". Note that in frequentist statistics, the sentence "Probability this is all luck" doesn't even make sense. Either the universe is so that it was luck, or it is not. In frequentist statistics you do not assign probabilities to models of the universe. You only assign probabilities to observations, given a fixed model of the universe.