4 ms·

Interesting post! A quibble: > If you’re a Bayesian, or use a model with confidence intervals baked in, you may be in pretty good shape. But let’s face it, Ba

by arcanus 10y ago

Interesting post! A quibble: > If you’re a Bayesian, or use a model with confidence intervals baked in, you may be in pretty good shape. But let’s face it, Bayesian techniques assume your prior is correct, and that new points are drawn from your prior. If your prior is wrong, so are your confidence intervals, and you have no way of knowing this. I don't agree. Baysian models must be validated, like any model. While no validation process is exhaustive, a predictive validation process is designed to directly test the applicability of the prior to a set of results. Furthermore, there are priors (Jeffrey's for example) that are entropy maximizing, from an information standpoint. These non-informative priors are designed to be used when an otherwise possibly misspecified prior would otherwise be introduced. It is not uncommon for reviewers to ask for results reproduced with Jeffries priors to ascertain of this is indeed the case.

damon_mcdougall 10y ago
Agreed.
stdbrouw 10y ago
... and in fact, it's machine learning models whose performance is tested using cross-validation or a holdout sample that assume new points are drawn from exactly the same distribution that generated the training data, whereas Bayesian priors are often used for the exact opposite purpose: when you're afraid the data is too noisy or unrepresentative and won't generalize without incorporating outside information. Also, there is no need to train any model whatsoever if your prior is correct. The prior is the model.
scottlocklin 10y ago
I said it in an awkward way. The important thing is, your CP gizmo will tell you when something has gone haywire while you're using it. Your Bayes doodad might not (other than noticing errors). In particular, some very important new point may have a bad prediction associated with it, with bad error bars: CP may be of big help here. There's an example of this in page 102-106 of the original book I think. Another good example is applying this idea to HMMs which you can read about in "Hidden Markov Models with Confidence" Giovanni Cherubin1 and Ilia Nouretdinov I'm not throwing poo at Bayesian models, which I think are sadly neglected these days, but with CP ideas you can get more useful results. While I think CP is useful for practitioners now, the most exciting applications are in stuff like active learning, and developing novel techniques associated with this basket of ideas. Really, my blog kind of missed the mark. I need to do more with examples.
- stdbrouw 10y ago
  > Really, my blog kind of missed the mark. It'd be sad if people only published stuff once they considered it perfect. I enjoyed the post, thanks for writing.
  - arcanus 10y ago
    > I enjoyed the post, thanks for writing. Same. My quibble was just part of the discussion, not intended to imply it was a poor article.

by arcanus 10y ago

damon_mcdougall 10y ago

Agreed.

stdbrouw 10y ago

... and in fact, it's machine learning models whose performance is tested using cross-validation or a holdout sample that assume new points are drawn from exactly the same distribution that generated the training data, whereas Bayesian priors are often used for the exact opposite purpose: when you're afraid the data is too noisy or unrepresentative and won't generalize without incorporating outside information. Also, there is no need to train any model whatsoever if your prior is correct. The prior is the model.

scottlocklin 10y ago

I said it in an awkward way. The important thing is, your CP gizmo will tell you when something has gone haywire while you're using it. Your Bayes doodad might not (other than noticing errors). In particular, some very important new point may have a bad prediction associated with it, with bad error bars: CP may be of big help here. There's an example of this in page 102-106 of the original book I think. Another good example is applying this idea to HMMs which you can read about in "Hidden Markov Models with Confidence" Giovanni Cherubin1 and Ilia Nouretdinov I'm not throwing poo at Bayesian models, which I think are sadly neglected these days, but with CP ideas you can get more useful results. While I think CP is useful for practitioners now, the most exciting applications are in stuff like active learning, and developing novel techniques associated with this basket of ideas. Really, my blog kind of missed the mark. I need to do more with examples.

stdbrouw 10y ago
> Really, my blog kind of missed the mark. It'd be sad if people only published stuff once they considered it perfect. I enjoyed the post, thanks for writing.
- arcanus 10y ago
  > I enjoyed the post, thanks for writing. Same. My quibble was just part of the discussion, not intended to imply it was a poor article.