3 ms·

> the problem is that random guessing (of priors) is not a reasonable replacement for science. I disagree! > After all, Bayes' rule will give nonsense answer

> the problem is that random guessing (of priors) is not a reasonable replacement for science. I disagree! > After all, Bayes' rule will give nonsense answers if you give it a nonsense prior! You are absolutely correct here, but draw the wrong conclusion. Bayes rule isn't just some convenience that you ought to ignore without a solid prior. Fundamentally, what can any study say about a subject whose prior likely ranges over a few orders of magnitude among the experts? That subject represents a gaping hole in our knowledge. If a single study were to narrow the band by a whole decade, it would still leave a decade sized hole... All of this should be explicit and out in the open. > I could imagine systemic worldwide ways to deal with this. For example, perhaps the scientific community could allow people to propose initial priors, and then allow multiple different papers to improve the estimation of the probability over time. Yeah, this is the way to go. Researchers should preregister their studies with their priors. Future studies will revise them. After all, its easy to argue about numbers and those who disagree can easily supply their own.

DataWorker 7y ago
This ignores the labor market economics involved in research. If there was lots of pre-registration and always multiple replication attempts then there wouldn’t be much of a replication crisis in the first place. It’s not frequentist stats that’s to blame, it’s the labor market for researchers.
- kansface 7y ago
  I appreciate your point, but surely both are problems? One is bad math while the other is a sort of a prisoners dilemma where everyone has already defected. I don't think changing the incentives would necessarily lead to better math or even that we have to change the incentives before changing the math.
- tomp 7y ago
  No, the problem is much wider than just researchers. If for example the public, or at least the media, didn’t buy the bullshit they were selling (with high p values, non-public data and other shoddy statistical hacks), there wouldn’t be any replication crisis either. The general problem is that just stamping the word “science” on a black box that smells of poo gives it much more credibility than it should, and it often also prevents people from thinking critically.

DataWorker 7y ago

This ignores the labor market economics involved in research. If there was lots of pre-registration and always multiple replication attempts then there wouldn’t be much of a replication crisis in the first place. It’s not frequentist stats that’s to blame, it’s the labor market for researchers.

kansface 7y ago
I appreciate your point, but surely both are problems? One is bad math while the other is a sort of a prisoners dilemma where everyone has already defected. I don't think changing the incentives would necessarily lead to better math or even that we have to change the incentives before changing the math.
tomp 7y ago
No, the problem is much wider than just researchers. If for example the public, or at least the media, didn’t buy the bullshit they were selling (with high p values, non-public data and other shoddy statistical hacks), there wouldn’t be any replication crisis either. The general problem is that just stamping the word “science” on a black box that smells of poo gives it much more credibility than it should, and it often also prevents people from thinking critically.