3 ms·
The problem is not so much the system as how it's presented: the media presents "peer-reviewed study" as a gold standard, practically synonymous with correct. A
by thu2111 6y ago
The problem is not so much the system as how it's presented: the media presents "peer-reviewed study" as a gold standard, practically synonymous with correct. And unfortunately academia absolutely encourages this practice.
The message that peer review is largely meaningless hasn't really been made by academia, for obvious reasons, and thus stupid papers getting accepted by journals will continue to be interesting and news-worthy until people get the message that peer review doesn't mean much.
The world will then move on to "replicated" as a gold standard. This will be better but not by much. Just a few weeks ago Imperial College London published a press release claiming their Report 9 results from their COVID-19 simulator program had been replicated. Worth a press release because after it was open sourced it was discovered to be filled with non-deterministic behaviour, even with fixed RNG seeds.
Unfortunately the press release was fraudulent. The report they cited as evidence of replication was by a friendly academic. He said he was able to replicate the results, then admitted every number he got out was different, some by 10%-25% different. This is the output of a computer simulation so the allowable difference is 0%. Despite this, Nature and other outlets proceeded to report that ICL's COVID model was "replicated".
In the end, academia is strongly incentivised to appear credible, not actually be so. Being credible requires you can reliably turn your findings into something useful, as when in the realm of pure theory you can never be sure your findings actually hold up in reality. Corporate research has this attribute (eventually), academic research doesn't, as academics are rewarded for writing clever sounding papers not being correct.
- tptacek 6y agoThis seems totally fair!