7 ms·

You should look at liblinear and Vowpal Wabbit. The former gives you a super-fast regularized logistic regression (and many other things). The later gives you o

You should look at liblinear and Vowpal Wabbit. The former gives you a super-fast regularized logistic regression (and many other things). The later gives you online classification, and you can fake a probabilistic model. On a related note, you're wasting clicks using A/B testing. I emailed you guys about using a better online method (a bandit algorithm) but never heard back from anyone. If that's of interest, drop me a line (noel at untyped dot com). Update It's occurred to me that you're using GAE, and so probably can't run C libraries like the above two projects. There is a Java library here: http://code.google.com/p/boxer-bayesian-regression/ http://code.google.com/p/boxer-bayesian-regression/ If you're going to do per user and per exercise models you'll have many fewer data points to train your models on. You should consider sharing data between models or use a model that will give some measure of uncertainty in it's predictions. The Bayesian LR code I referenced above will give some measure of uncertainty. There is a stack of (really interesting) work on other methods that will also do this.

brown9-2 15y ago
I have nothing to do with KA but I am curious by what you mean about A/B testing and bandit algorithms. Would you mind elaborating or sharing a link or two?
- noelwelsh 15y ago
  I'm totally pimping my own warez here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m... http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html
  - Eliezer 15y ago
    (Reads links.) I've been going around telling people for a while that A/B testing is non-Bayesian but I didn't realize there was an off-the-shelf solution! You need to pimp your wares more often.
    - ced 15y ago
      It doesn't sound terribly Bayesian to me. From one of his pages: However, the significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance. If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. The original intent of the experimenter shouldn't affect his conclusions! With that said, I'm not sure what the proper Bayesian answer is. If I were to stop my experiments every time that P(X|D) < 0.05 (or whatever), wouldn't that introduce bias? [1] Incidentally, you (Eliezer) had a nice post where you explained the problem in detail (something about two doctors doing an experiment 50 times with different "stopping rules", who get the same data), but I couldn't find it. [1] EDIT: On further thought, there wouldn't be any bias if I do the correct thing and report my uncertainty: P(X|D) = 0.04. The "repeated significance testing errors" comes from converting P = 0.04 into an absolute statement "X is true". Then the stopping rule will affect the statistics.
      - genieyclo 15y ago
        Is this it? http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes
        ced 15y ago
        Nah, it's not. The original had a setup like: Doctor A decides to test a cure on 50 patients. 40 have gotten better. Doctor B independently decides to test the same cure on his patients. He will stop once he has reached 'significance'. Coincidentally, the results become significant at the 50th patient, and he also has a 40/50 success rate. Doctor A says "I followed a fixed testing procedure, and the statistical analysis says that my data is not significant. We need more experiments." Doctor B says "I followed an optional stopping procedure, and the statistical analysis says that my data is not significant: the cure is good." A Bayesian would claim that if they both have the same data, then they should reach the same conclusion, regardless of their intent. A "frequentist" would uphold that the doctors can legitimately disagree. I don't know much about frequentism, but it's the dominant perspective in statistics. Everything I've read about A/B testing is frequentist.

brown9-2 15y ago

I have nothing to do with KA but I am curious by what you mean about A/B testing and bandit algorithms. Would you mind elaborating or sharing a link or two?

noelwelsh 15y ago
I'm totally pimping my own warez here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-make-out-like-a-bandit/ http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m... http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html
- Eliezer 15y ago
  (Reads links.) I've been going around telling people for a while that A/B testing is non-Bayesian but I didn't realize there was an off-the-shelf solution! You need to pimp your wares more often.
  - ced 15y ago
    It doesn't sound terribly Bayesian to me. From one of his pages: However, the significance calculation makes a critical assumption that you have probably violated without even realizing it: that the sample size was fixed in advance. If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. The original intent of the experimenter shouldn't affect his conclusions! With that said, I'm not sure what the proper Bayesian answer is. If I were to stop my experiments every time that P(X|D) < 0.05 (or whatever), wouldn't that introduce bias? [1] Incidentally, you (Eliezer) had a nice post where you explained the problem in detail (something about two doctors doing an experiment 50 times with different "stopping rules", who get the same data), but I couldn't find it. [1] EDIT: On further thought, there wouldn't be any bias if I do the correct thing and report my uncertainty: P(X|D) = 0.04. The "repeated significance testing errors" comes from converting P = 0.04 into an absolute statement "X is true". Then the stopping rule will affect the statistics.
    - genieyclo 15y ago
      Is this it? http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes
      - ced 15y ago
        Nah, it's not. The original had a setup like: Doctor A decides to test a cure on 50 patients. 40 have gotten better. Doctor B independently decides to test the same cure on his patients. He will stop once he has reached 'significance'. Coincidentally, the results become significant at the 50th patient, and he also has a 40/50 success rate. Doctor A says "I followed a fixed testing procedure, and the statistical analysis says that my data is not significant. We need more experiments." Doctor B says "I followed an optional stopping procedure, and the statistical analysis says that my data is not significant: the cure is good." A Bayesian would claim that if they both have the same data, then they should reach the same conclusion, regardless of their intent. A "frequentist" would uphold that the doctors can legitimately disagree. I don't know much about frequentism, but it's the dominant perspective in statistics. Everything I've read about A/B testing is frequentist.