6 ms·

> The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment woul

> The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help. The strength of significance testing is that it purposely doesn't try to tell you how likely something is to be true, only how likely the data you got was the result of chance assuming the treatment is no better than placebo. You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. My concern with bayesian approaches is that, like with frequentist approaches, the truth is still fundamentally unknowable, only now you're encouraged to put a number on that and pretend that it's science. While bayesian approaches totally make sense in trying to determine a patient's likelihood of having some disease when there is already data available for the prevalence in a population and the sensitivity and specificity of the tests, using bayesian logic to weight clinical trials strikes me as being highly dubious. It would be one thing if SBM actually developed a framework to give a weight to each methodological feature of a trial, but so far I haven't seen much work to build a functioning system. Though if you're really honest about all the ways that you can have positive results without something actually being true, it seems like almost no amount of research will ever have a significant effect on the prior.

mdbco 12y ago
You're absolutely correct that Bayesian approaches are not magical and do not suddenly supply you with vastly more information than frequentist approaches (particularly when you have a really poor prior, in which case the Bayesian approach will be similarly poor). Bayesian statistics is certainly very popular right now, but it should not be looked at as some sort of panacea for all statistical problems. However, I would say that Bayesian approaches do have a big advantage in terms of helping with the interpretation problems that plague frequentist significance testing. Namely, as the OP article points out, Bayesian approaches reformulate the testing question in a way that is more intuitive, i.e. "what is the probability of the hypothesis given both the prior probability and the new data?". So yes, Bayesian methods surely do not fix everything, but since interpretation of statistics is such a major concern, they can be quite beneficial.
yummyfajitas 12y ago
You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. This is a danger sign - you are doing the same things the Bayesians do, just informally, less explicitly, and probably incorrectly. The fact is that to make a good decision, eventually you need to compute a single number. This is an elementary fact of topology: https://www.chrisstucchio.com/blog/2014/topology_of_decisionmaking.html https://www.chrisstucchio.com/blog/2014/topology_of_decision... That number will be based on some unproveable assumptions. That's a fact of Godel's incompleteness theorem, if nothing else. So given this, why is it "dubious" to make those assumptions explicit and obvious?
- algorias 12y ago
  > The fact is that to make a good decision, eventually you need to compute a single number. Your linked blog post states that if you make a good decision, then there is a process computing a single number which is equivalent to your process. This is not equivalent to what you claim. As a matter of fact, it's the same kind of confusion that exists around the p-value. It's not the case that a process explicitly computing such a number automatically makes good decisions, which is what you seem to claim implicitly. Also, Gödel has nothing whatsoever to to with this.
  - yummyfajitas 12y ago
    Eventually you need to compute a number which is either above or below your go/no go threshold. That's the number I'm referring to. I don't claim you can't arrive at it by some perfect heuristic. I merely claim that you are better off being explicit about your assumptions and formalizing your reasoning. That just makes mistakes more obvious, makes your strong assumptions more clear, and makes it more likely that you will correctly update your beliefs rather than incorrectly discounting/overvaluing evidence. You are right about godel, it's a separate theorem I'm referring to which says you need unproveable axioms. I misremembered, sorry, wrote that before my coffee.
    - algorias 12y ago

mdbco 12y ago

You're absolutely correct that Bayesian approaches are not magical and do not suddenly supply you with vastly more information than frequentist approaches (particularly when you have a really poor prior, in which case the Bayesian approach will be similarly poor). Bayesian statistics is certainly very popular right now, but it should not be looked at as some sort of panacea for all statistical problems. However, I would say that Bayesian approaches do have a big advantage in terms of helping with the interpretation problems that plague frequentist significance testing. Namely, as the OP article points out, Bayesian approaches reformulate the testing question in a way that is more intuitive, i.e. "what is the probability of the hypothesis given both the prior probability and the new data?". So yes, Bayesian methods surely do not fix everything, but since interpretation of statistics is such a major concern, they can be quite beneficial.

yummyfajitas 12y ago

You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it. This is a danger sign - you are doing the same things the Bayesians do, just informally, less explicitly, and probably incorrectly. The fact is that to make a good decision, eventually you need to compute a single number. This is an elementary fact of topology: https://www.chrisstucchio.com/blog/2014/topology_of_decisionmaking.html https://www.chrisstucchio.com/blog/2014/topology_of_decision... That number will be based on some unproveable assumptions. That's a fact of Godel's incompleteness theorem, if nothing else. So given this, why is it "dubious" to make those assumptions explicit and obvious?

algorias 12y ago
> The fact is that to make a good decision, eventually you need to compute a single number. Your linked blog post states that if you make a good decision, then there is a process computing a single number which is equivalent to your process. This is not equivalent to what you claim. As a matter of fact, it's the same kind of confusion that exists around the p-value. It's not the case that a process explicitly computing such a number automatically makes good decisions, which is what you seem to claim implicitly. Also, Gödel has nothing whatsoever to to with this.
- yummyfajitas 12y ago
  Eventually you need to compute a number which is either above or below your go/no go threshold. That's the number I'm referring to. I don't claim you can't arrive at it by some perfect heuristic. I merely claim that you are better off being explicit about your assumptions and formalizing your reasoning. That just makes mistakes more obvious, makes your strong assumptions more clear, and makes it more likely that you will correctly update your beliefs rather than incorrectly discounting/overvaluing evidence. You are right about godel, it's a separate theorem I'm referring to which says you need unproveable axioms. I misremembered, sorry, wrote that before my coffee.
  - algorias 12y ago