7 ms·

I have nothing to do with KA but I am curious by what you mean about A/B testing and bandit algorithms. Would you mind elaborating or sharing a link or two?

noelwelsh 15y ago
I'm totally pimping my own warez here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m... http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html
- Eliezer 15y ago
  (Reads links.) I've been going around telling people for a while that A/B testing is non-Bayesian but I didn't realize there was an off-the-shelf solution! You need to pimp your wares more often.
  - ced 15y ago
    It doesn't sound terribly Bayesian to me. From one of his pages: However, the significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance. If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. The original intent of the experimenter shouldn't affect his conclusions! With that said, I'm not sure what the proper Bayesian answer is. If I were to stop my experiments every time that P(X|D) < 0.05 (or whatever), wouldn't that introduce bias? [1] Incidentally, you (Eliezer) had a nice post where you explained the problem in detail (something about two doctors doing an experiment 50 times with different "stopping rules", who get the same data), but I couldn't find it. [1] EDIT: On further thought, there wouldn't be any bias if I do the correct thing and report my uncertainty: P(X|D) = 0.04. The "repeated significance testing errors" comes from converting P = 0.04 into an absolute statement "X is true". Then the stopping rule will affect the statistics.
    - genieyclo 15y ago
      Is this it? http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes
      - ced 15y ago
        Nah, it's not. The original had a setup like: Doctor A decides to test a cure on 50 patients. 40 have gotten better. Doctor B independently decides to test the same cure on his patients. He will stop once he has reached 'significance'. Coincidentally, the results become significant at the 50th patient, and he also has a 40/50 success rate. Doctor A says "I followed a fixed testing procedure, and the statistical analysis says that my data is not significant. We need more experiments." Doctor B says "I followed an optional stopping procedure, and the statistical analysis says that my data is not significant: the cure is good." A Bayesian would claim that if they both have the same data, then they should reach the same conclusion, regardless of their intent. A "frequentist" would uphold that the doctors can legitimately disagree. I don't know much about frequentism, but it's the dominant perspective in statistics. Everything I've read about A/B testing is frequentist.
        arundelo 15y ago
        Sounds like this: http://lesswrong.com/lw/mt/beautiful_probability/ http://lesswrong.com/lw/mt/beautiful_probability/
        ced 15y ago
        Yes! Thank you.
    - xenophanes 15y ago
      Changing stopping rules after seeing the data creates bias/distortions. One does have to set the rules in advance, that's not a mistake. Consider a data set consisting of a single license plate number. (This example is from Richard Feynman.) If you set the rules in retrospect, you can go "Wow, what were the odds my one license plate would be XJKDL-2342-KE? One in a million?" But that's wrong. On the other hand if you predicted XJKDL-2342-KE in advance, then the same data point would have a different meaning. How did you predict it? Patterns that you can predict in advance are different from ones you can find in retrospect after looking through whatever results you get. So the same data point -- XJKDL-2342-KE -- can take on different meaning depending on the original intent and design of the experimenter. People make this mistake all the time with more mundane examples. Like they will roll snake eyes three times in a row, then calculate the odds of that happening, and then says "wow 1/6^6, there was such a minuscule chance i'd get screwed like this". but they're just wrong. ANY exact ordering of the 6 individual dice roles has a 1/6^6 chance of happening, and you have to get one of the "unlikely" results. To help make this more intuitive, consider that they would have been surprised by rolling all 2s, or all 3s, or various other patterns. So you at least would have to figure out how many outcomes they'd deem surprising and figure out what proportion of the possibilities are in that category. And then take into account all the rolls they made when they weren't surprised and didn't record any data...

noelwelsh 15y ago

I'm totally pimping my own warez here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m... http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html

Eliezer 15y ago
(Reads links.) I've been going around telling people for a while that A/B testing is non-Bayesian but I didn't realize there was an off-the-shelf solution! You need to pimp your wares more often.
- ced 15y ago
  It doesn't sound terribly Bayesian to me. From one of his pages: However, the significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance. If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. The original intent of the experimenter shouldn't affect his conclusions! With that said, I'm not sure what the proper Bayesian answer is. If I were to stop my experiments every time that P(X|D) < 0.05 (or whatever), wouldn't that introduce bias? [1] Incidentally, you (Eliezer) had a nice post where you explained the problem in detail (something about two doctors doing an experiment 50 times with different "stopping rules", who get the same data), but I couldn't find it. [1] EDIT: On further thought, there wouldn't be any bias if I do the correct thing and report my uncertainty: P(X|D) = 0.04. The "repeated significance testing errors" comes from converting P = 0.04 into an absolute statement "X is true". Then the stopping rule will affect the statistics.
  - genieyclo 15y ago
    Is this it? http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes
    - ced 15y ago
      Nah, it's not. The original had a setup like: Doctor A decides to test a cure on 50 patients. 40 have gotten better. Doctor B independently decides to test the same cure on his patients. He will stop once he has reached 'significance'. Coincidentally, the results become significant at the 50th patient, and he also has a 40/50 success rate. Doctor A says "I followed a fixed testing procedure, and the statistical analysis says that my data is not significant. We need more experiments." Doctor B says "I followed an optional stopping procedure, and the statistical analysis says that my data is not significant: the cure is good." A Bayesian would claim that if they both have the same data, then they should reach the same conclusion, regardless of their intent. A "frequentist" would uphold that the doctors can legitimately disagree. I don't know much about frequentism, but it's the dominant perspective in statistics. Everything I've read about A/B testing is frequentist.
      - arundelo 15y ago
        Sounds like this: http://lesswrong.com/lw/mt/beautiful_probability/ http://lesswrong.com/lw/mt/beautiful_probability/
        ced 15y ago
        Yes! Thank you.
  - xenophanes 15y ago
    Changing stopping rules after seeing the data creates bias/distortions. One does have to set the rules in advance, that's not a mistake. Consider a data set consisting of a single license plate number. (This example is from Richard Feynman.) If you set the rules in retrospect, you can go "Wow, what were the odds my one license plate would be XJKDL-2342-KE? One in a million?" But that's wrong. On the other hand if you predicted XJKDL-2342-KE in advance, then the same data point would have a different meaning. How did you predict it? Patterns that you can predict in advance are different from ones you can find in retrospect after looking through whatever results you get. So the same data point -- XJKDL-2342-KE -- can take on different meaning depending on the original intent and design of the experimenter. People make this mistake all the time with more mundane examples. Like they will roll snake eyes three times in a row, then calculate the odds of that happening, and then says "wow 1/6^6, there was such a minuscule chance i'd get screwed like this". but they're just wrong. ANY exact ordering of the 6 individual dice roles has a 1/6^6 chance of happening, and you have to get one of the "unlikely" results. To help make this more intuitive, consider that they would have been surprised by rolling all 2s, or all 3s, or various other patterns. So you at least would have to figure out how many outcomes they'd deem surprising and figure out what proportion of the possibilities are in that category. And then take into account all the rolls they made when they weren't surprised and didn't record any data...