3 ms·

From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct. On the other hand, the Bayesian app

From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct. On the other hand, the Bayesian approach is to split the data into training and test sets, and verify how much of it holds. Does this make sense? If the p-values are not very good at conveying "confidence in the prediction," is this another argument in favor of a more Bayesian approach to statistics? Any thoughts would be appreciated!

gbrown 11y ago
Well, neither your description of p-values, nor your understanding of Bayesian inference is correct, so... no?
- TeMPOraL 11y ago
  It would be helpful to tell how the commenter is wrong, and what is the correct description.
  - gbrown 11y ago
    It would also be helpful if they'd read the article.
    - TeMPOraL 11y ago
      I read the article and it's almost completely orthogonal to the questions asked by 'graffitici. There's nothing suggesting that they didn't read the article, so now I assert that you have three things to explain - where they're wrong, what's the right answer to their question and why on Earth did you think they didn't read the article?
      - gbrown 11y ago
        From the article: "We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance." From the first sentence of the parent comment: "From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct." If my reply seemed glib, it was because I thought the question was so far off base, that graffitici almost certainly hadn't read the article. To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above. If there was a simple statistical misunderstanding, perhaps that would be worth clearing up. Instead, the parent comment used some statistical words, but was very nearly incoherent. Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.
        TeMPOraL 11y ago
        > To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above. I disagree. There's an alternative hypothesis that could explain a comment like that - the author harbors some confused views about frequentist and Bayesian approaches to statistics. I'm inclined to believe that this hypothesis is right, because I recognize that comment as something I'd write myself back when I was more confused about this topic. Reading the article is unlikely to affect this particular issue. > Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference. I think even one sentence explaining the gist of the author's confusion would be enough. Plenty of essays have been written on the topic, but one needs to be pointed in their general direction in order to benefit.
        gbrown 11y ago
        That's a fair point, but I'm not sure I understand the gist of the author's confusion. Any response I would craft would end up being a re-statement of the usual one sentence definition of p-values and Bayesian probability, both of which are already under discussion elsewhere in the comments.
peatmoss 11y ago
This isn't really correct on either point. A p-value is how a frequentist establishes "statistical significance," which is itself convoluted. It has really nothing to do with validation. On the second point, frequentists very commonly split training and testing datasets, and so this practice is pretty much orthogonal to whether you're using Bayesian or frequentist methods.
SEMW 11y ago
> the Bayesian approach is to split the data into training and test sets, and verify how much of it holds. The Bayesian approach would be to say that the thing you generally want to know isn't {the probability of getting this result by chance if the hypothesis is false}, it's the probability the hypothesis is true, given everything you know up to and including the new data. So they would calculate p(hypothesis is true given the new data) using Bayes theorem -- which requires inputting what they thought p(hypothesis is true) was at the start of the experiment, before the new data came in. http://www.yudkowsky.net/rational/bayes http://www.yudkowsky.net/rational/bayes is a decent explanation.
KingMob 11y ago
In practice, people, even scientists, treat smaller p-values as better "evidence". That's not what p-vals mean, and even if it were, there are biases in p-vals that make them problematic as the number of data points increase. Bayesian stats aren't related to training and test sets, like in machine learning; scientific experiments rarely partition data like that. A Bayesian analysis answers the question, "Given the evidence I just collected, how should I adjust my estimate of how likely something is?" But Bayesian analyses have issues too: the biggest is not knowing what your initial, "prior" probability should be. (After all, that's usually what you're trying to find out!) With something simple like a coin flip, you have a strong prior assumption of 50% probability of heads. And neither Bayesian nor frequentist methods address effect size. If you collect enough data, you can detect extremely small (real) effect sizes that pass statistical tests, but are still meaningless outside the context of publishing a paper. Rather than analysis type, we should incorporate more discussion of effect size, especially as the number of data points increase.

gbrown 11y ago

Well, neither your description of p-values, nor your understanding of Bayesian inference is correct, so... no?

TeMPOraL 11y ago
It would be helpful to tell how the commenter is wrong, and what is the correct description.
- gbrown 11y ago
  It would also be helpful if they'd read the article.
  - TeMPOraL 11y ago
    I read the article and it's almost completely orthogonal to the questions asked by 'graffitici. There's nothing suggesting that they didn't read the article, so now I assert that you have three things to explain - where they're wrong, what's the right answer to their question and why on Earth did you think they didn't read the article?
    - gbrown 11y ago
      From the article: "We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance." From the first sentence of the parent comment: "From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct." If my reply seemed glib, it was because I thought the question was so far off base, that graffitici almost certainly hadn't read the article. To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above. If there was a simple statistical misunderstanding, perhaps that would be worth clearing up. Instead, the parent comment used some statistical words, but was very nearly incoherent. Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.
      - TeMPOraL 11y ago
        > To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above. I disagree. There's an alternative hypothesis that could explain a comment like that - the author harbors some confused views about frequentist and Bayesian approaches to statistics. I'm inclined to believe that this hypothesis is right, because I recognize that comment as something I'd write myself back when I was more confused about this topic. Reading the article is unlikely to affect this particular issue. > Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference. I think even one sentence explaining the gist of the author's confusion would be enough. Plenty of essays have been written on the topic, but one needs to be pointed in their general direction in order to benefit.
        gbrown 11y ago
        That's a fair point, but I'm not sure I understand the gist of the author's confusion. Any response I would craft would end up being a re-statement of the usual one sentence definition of p-values and Bayesian probability, both of which are already under discussion elsewhere in the comments.

peatmoss 11y ago

This isn't really correct on either point. A p-value is how a frequentist establishes "statistical significance," which is itself convoluted. It has really nothing to do with validation. On the second point, frequentists very commonly split training and testing datasets, and so this practice is pretty much orthogonal to whether you're using Bayesian or frequentist methods.

SEMW 11y ago

> the Bayesian approach is to split the data into training and test sets, and verify how much of it holds. The Bayesian approach would be to say that the thing you generally want to know isn't {the probability of getting this result by chance if the hypothesis is false}, it's the probability the hypothesis is true, given everything you know up to and including the new data. So they would calculate p(hypothesis is true given the new data) using Bayes theorem -- which requires inputting what they thought p(hypothesis is true) was at the start of the experiment, before the new data came in. http://www.yudkowsky.net/rational/bayes http://www.yudkowsky.net/rational/bayes is a decent explanation.

KingMob 11y ago

In practice, people, even scientists, treat smaller p-values as better "evidence". That's not what p-vals mean, and even if it were, there are biases in p-vals that make them problematic as the number of data points increase. Bayesian stats aren't related to training and test sets, like in machine learning; scientific experiments rarely partition data like that. A Bayesian analysis answers the question, "Given the evidence I just collected, how should I adjust my estimate of how likely something is?" But Bayesian analyses have issues too: the biggest is not knowing what your initial, "prior" probability should be. (After all, that's usually what you're trying to find out!) With something simple like a coin flip, you have a strong prior assumption of 50% probability of heads. And neither Bayesian nor frequentist methods address effect size. If you collect enough data, you can detect extremely small (real) effect sizes that pass statistical tests, but are still meaningless outside the context of publishing a paper. Rather than analysis type, we should incorporate more discussion of effect size, especially as the number of data points increase.