3 ms·

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst

dalbasal 5y ago
True, and a lot of it is fraud or not far. However, a big chunk of this statistic is not actually fraud. It's just false empiricism. If you look at a lot of the actual papers, I think you'll find that a lot of them just aren't trying to do empiricism... at least not strict empiricism. They're forced into that structure by a convention. Either (1) "Not Science" becomes OK and extends to economics, some areas of ecology, and such. (2) The Scientific Method gets renamed "Method A," and the concept of scientific pursuit is more flexible, depending on what you are studying or (3) We force these poor bastards doing economics, psychology and such to pretend that they're doing a method that they're not... or publish in the prestigious "Unscientific Economics & Psychology" journal.
tpoacher 5y ago
It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this: > Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’
- deleted 5y ago
  [deleted]

dalbasal 5y ago

True, and a lot of it is fraud or not far. However, a big chunk of this statistic is not actually fraud. It's just false empiricism. If you look at a lot of the actual papers, I think you'll find that a lot of them just aren't trying to do empiricism... at least not strict empiricism. They're forced into that structure by a convention. Either (1) "Not Science" becomes OK and extends to economics, some areas of ecology, and such. (2) The Scientific Method gets renamed "Method A," and the concept of scientific pursuit is more flexible, depending on what you are studying or (3) We force these poor bastards doing economics, psychology and such to pretend that they're doing a method that they're not... or publish in the prestigious "Unscientific Economics & Psychology" journal.

tpoacher 5y ago

It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this: > Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’

deleted 5y ago
[deleted]