4 ms·

But I don't see p-values as connected to frequentist statistics. Rather, p-values are a concept that applies to the problem of hypothesis testing, which both fr

by mjfl 3y ago

But I don't see p-values as connected to frequentist statistics. Rather, p-values are a concept that applies to the problem of hypothesis testing, which both frequentist and bayesian statistics should apply to, right? And in both cases you would use a p-value. The unique thing that Bayesian statistics can do is fit parameters can calculate uncertainty in those parameters, which does not contradict any of the concepts of hypothesis testing.

travisjungroth 3y ago
A p-value is the frequency you’d expect to see a sample with an effect this size or greater, given the null hypothesis. It’s inherently a frequentist approach to inference. You can squish it into Bayes by considering it a uniform prior on the real number line that you never update, but you’re not really doing things “in the spirit” of Bayes, then.
- mjfl 3y ago
  What is a “prior” in the context of hypothesis testing? That seems to me to be a concept used in parameter estimation, with Bayes rule. But hypothesis testing is not parameter estimation.
  - travisjungroth 3y ago
    As the linked post points out, you can regularize a p-value based on prior experiments. IMO, null hypothesis testing is way overused. We should be quantifying the comparison of the null versus alternative hypotheses. People already compare them. Might as well bring some math into it.
  - mitthrowaway2 3y ago
    How is hypothesis testing different from parameter estimation? They seem conceptually the same to me, with one a special case of the other.
  - PheonixPharts 3y ago
    In Bayesian terms there is no distinction between hypothesis testing and parameter estimation. If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem. You can consider the distribution of estimates for a parameter to represent a space of possible hypothesis for the true value of that parameter and their absolutely or relative likelihood based on the information available. The also has the pleasant consequence that hypothesis testing using Bayesian methods nearly always is closer to the actual question someone wants to answer rather than replying with confusing statements such as "the probability of rejecting the null hypothesis", which contains an implicit double negative.
    - mjfl 3y ago
      > If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem. but even in this situation you would need a threshold for decision making, i.e. a p-value.
      - uoaei 3y ago
        "p-value" as a term already has a definition -- one that imposes a frequentist perspective -- so you'd have to come up with a new term to define a Bayesian analogue of that concept.
        mjfl 3y ago
        I fail to understand why the terminology matters. If you need to make a decision then there will be a threshold in the probability that will determine it. It doesn’t matter if you call it p values or anything else.
        uoaei 3y ago
        I'm not understanding your confusion. Different concepts require different words to distinguish them in conversation. p-value is a name, not merely a descriptor, of one possible way to define a threshold. Other ways of defining thresholds get different names so that everyone knows what's being discussed. Bob bob bob'bob bob bob bob bob, bob bob bob bob bob "Bob".
  - stale2002 3y ago
    > What is a “prior” in the context of hypothesis testing? A prior could be any previous knowledge that you have on the subject, or anything about the world. Let me give you a silly example. Imagine I do an experiment, on if person A can see the future, and predict coin flips. Now, imagine that I do this experiment, and lo and behold it passes the 5% threshold P value! Remember what a P value is. It is simply the percent chance that the results that you got were by chance. So, if I do that experiment, and it actually gets at the "P value" of 5%, which means that there is a 5% chance of it happening by chance, do you know believe that Person A can see the future? I wouldn't. I would need a much lower P value to start believing that. Thats all a prior is. It is using the fact that "Person A can see the future!" is an extraordinary claim. And for that, we require extraordinary evidence. Much better evidence than a mere "5% that it happened by chance". P
tingletech 3y ago
I think with a Bayesian test one would use credible intervals rather than p values?
PheonixPharts 3y ago
> But I don't see p-values as connected to frequentist statistics. P-values are fundamentally Frequentist because this framework see that observed data as random, whereas Bayesian statistics believe our observations are the only thing that is actually known, and all the other parameters are what is random in our experiments. That is: the data is the only part of an experiment that is real, everything else that we can't directly observe is something we must hypothesize about.

by mjfl 3y ago

travisjungroth 3y ago

A p-value is the frequency you’d expect to see a sample with an effect this size or greater, given the null hypothesis. It’s inherently a frequentist approach to inference. You can squish it into Bayes by considering it a uniform prior on the real number line that you never update, but you’re not really doing things “in the spirit” of Bayes, then.

mjfl 3y ago
What is a “prior” in the context of hypothesis testing? That seems to me to be a concept used in parameter estimation, with Bayes rule. But hypothesis testing is not parameter estimation.
- travisjungroth 3y ago
  As the linked post points out, you can regularize a p-value based on prior experiments. IMO, null hypothesis testing is way overused. We should be quantifying the comparison of the null versus alternative hypotheses. People already compare them. Might as well bring some math into it.
- mitthrowaway2 3y ago
  How is hypothesis testing different from parameter estimation? They seem conceptually the same to me, with one a special case of the other.
- PheonixPharts 3y ago
  In Bayesian terms there is no distinction between hypothesis testing and parameter estimation. If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem. You can consider the distribution of estimates for a parameter to represent a space of possible hypothesis for the true value of that parameter and their absolutely or relative likelihood based on the information available. The also has the pleasant consequence that hypothesis testing using Bayesian methods nearly always is closer to the actual question someone wants to answer rather than replying with confusing statements such as "the probability of rejecting the null hypothesis", which contains an implicit double negative.
  - mjfl 3y ago
    > If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem. but even in this situation you would need a threshold for decision making, i.e. a p-value.
    - uoaei 3y ago
      "p-value" as a term already has a definition -- one that imposes a frequentist perspective -- so you'd have to come up with a new term to define a Bayesian analogue of that concept.
      - mjfl 3y ago
        I fail to understand why the terminology matters. If you need to make a decision then there will be a threshold in the probability that will determine it. It doesn’t matter if you call it p values or anything else.
        uoaei 3y ago
        I'm not understanding your confusion. Different concepts require different words to distinguish them in conversation. p-value is a name, not merely a descriptor, of one possible way to define a threshold. Other ways of defining thresholds get different names so that everyone knows what's being discussed. Bob bob bob'bob bob bob bob bob, bob bob bob bob bob "Bob".
- stale2002 3y ago
  > What is a “prior” in the context of hypothesis testing? A prior could be any previous knowledge that you have on the subject, or anything about the world. Let me give you a silly example. Imagine I do an experiment, on if person A can see the future, and predict coin flips. Now, imagine that I do this experiment, and lo and behold it passes the 5% threshold P value! Remember what a P value is. It is simply the percent chance that the results that you got were by chance. So, if I do that experiment, and it actually gets at the "P value" of 5%, which means that there is a 5% chance of it happening by chance, do you know believe that Person A can see the future? I wouldn't. I would need a much lower P value to start believing that. Thats all a prior is. It is using the fact that "Person A can see the future!" is an extraordinary claim. And for that, we require extraordinary evidence. Much better evidence than a mere "5% that it happened by chance". P

tingletech 3y ago

I think with a Bayesian test one would use credible intervals rather than p values?

PheonixPharts 3y ago

> But I don't see p-values as connected to frequentist statistics. P-values are fundamentally Frequentist because this framework see that observed data as random, whereas Bayesian statistics believe our observations are the only thing that is actually known, and all the other parameters are what is random in our experiments. That is: the data is the only part of an experiment that is real, everything else that we can't directly observe is something we must hypothesize about.