4 ms·
Short, good read. The money quote is near the end: > We also suggest a way of powering future experiments: power your experiment such that you, or someone else
by StefanKarpinski 10y ago
Short, good read. The money quote is near the end:
> We also suggest a way of powering future experiments: power your experiment such that you, or someone else, can conduct a similarly-sized experiment and have high power for detecting an interesting difference from your study. We need to stop thinking about studies as if they are one-offs, only to be interpreted once in light of the hypotheses of the original authors.
- tpeo 10y agoIsn't this simply maximizing the probability of a successful reproduction? Seems almost obvious given the whole problem was low reproducibility in the first place.
- deegles 10y agoI thought that was what we wanted?
- SilasX 10y agoSufficiently advanced reproducibility is indistinguishable from correctness.
- TheOtherHobbes 10y agoOr systemic (as opposed to systematic) bias. Independent, adversarial reproducibility implies some possibility of correctness. But psychology doesn't seem to do what science usually does - build models and make independent predictions from them. Finding patterns in data is not the same as testing a null hypothesis which is the result of a model, and not just a random query. Or an accidental discovery.
- bobbles 10y agoSeems like there should be a ranking for verification. Something like 'paper accepted but not replicated' being rank 1, then replicated once gets you to level 2, etc. til you hit 5+ 'validity rating'. Might actually incentivize people then to be the 'replicators'
- jsprogrammer 10y agoThat is essentially the function of p-values.
- jsprogrammer 10y agoEdit time ran out: If you can't dispute it, you won't.
- Smaug123 10y agoI'll bite: p-values attempt to measure "Given the null hypothesis, what is the probability that we obtain a more extreme result than we actually did?". What we actually want is "What is the probability of the null hypothesis given the data?". If we are to use p-values successfully to determine truth, then by Bayes we must use prior probabilities somewhere. Discovering p=0.01 is not very helpful when your prior probability was 0.001.
- jsprogrammer 10y agoAny valid experiment would assume a prior p-value of 1 (otherwise, what are you hypothesizing?). A null hypothesis does not have a probability. It is either the case, or it isn't. Edit (response to your comment below) [speech restricted at the moment]: "If you can't dispute it, you won't." Edit2 (response to your comment/edit below): It's just a fact. Same as the other simple facts I have stated that remain completely undisputed. Why would you hypothesize an experiment where you already biased yourself (I assume p = 0.01? what?) on the outcome? If your hypothesis is that you will observe no differences, then you have implicitly assigned a prior of 1. Any experimental design will be reducible to this fundamental comparison. The p-value of an ongoing experiment is a measurement of the reproducibility of the [measurement of the predicted observations of the] hypothesis (many hypotheses are not true though; in those cases the p-values are not valid reproducibility measurements). Edit3: >Your misunderstanding in a nutshell is demonstrated by "if you can't dispute it, you won't", which you seem to think means "if you won't dispute it, you can't", in clear violation of Bayes's rule. I think that that is your misunderstanding.