3 ms·

How do you measure consensus? There are many researchers who claim there's a consensus of whatever they happen to believe, but when counter-examples are pointed

How do you measure consensus? There are many researchers who claim there's a consensus of whatever they happen to believe, but when counter-examples are pointed out they start No True Scotsmanning ("no real expert believes..."). Health related research is in a much worse state than ML unfortunately. ML suffers from metrics gaming and overfitting but there's probably not much outright fraud? Common estimates are that half of all medical papers are making false claims. This blog post of mine from a few years ago has some quotes from editors of well known medical journals where they express disbelief in their own published research base, and some examples of blatant fraud [0]. And this interview with Marc Andreessen indicates why VCs don't invest into biotech startups much [1]: I had a conversation with the long-time head of one of the big federal funding agencies for healthcare research who is also a very accomplished entrepreneur, and I said, “do you really think it’s true that 50-70% of biomedical research is fake?” This is a guy who has spent his life in this world. And he said “oh no, that’s not true at all. It’s 90%.” [Richard laughs]. I was like “holy shit,” I was flabbergasted that it could be 90%. [...] I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. In the same interview Andreessen references a study [2] that implies most medical/drug research stopped working overnight in the year 2000. The reason being that in that year the US govt started requiring drug trials to pre-register their hypothesis and then evaluate their results relative to that hypothesis i.e. they tried to prevent P-hacking. We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease [...] 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 [0] https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 [1] https://www.richardhanania.com/p/flying-x-wings-into-the-death-star https://www.richardhanania.com/p/flying-x-wings-into-the-dea... [2] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382&utm_content=buffer2135a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer https://journals.plos.org/plosone/article?id=10.1371/journal...

DrScientist 3y ago
> well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Odd that tenure is singled out as the core of the problem. I'd say the undermining of tenure is a large part of the problem. Research is hard - very hard - you are being asked to discover stuff nobody else has - to push the boundaries of knowledge. I would argue that the idea that one failed research project, and you are out on your ear because of a gap in publishing - is a core part of the problem. In terms of medical doctors doing research - I'd argue that's a special dangerous mix - medical doctors need to have an element of self-belief just to be able to do their job ( life and death decisions ). Instilling it is part of the training. That self-belief, plus a lack ( in most cases ) of any in depth scientific research training is a dangerous mix. Medical training != science training. In this specific case I have no idea about the rights and wrongs.
- pas 3y ago
  Research might be hard, harder than whatever else the rest of the world is doing, but the tenure system is not giving us the best researchers. It rewards those that are at the right place at the right time, it's mostly luck of the current spectacle, and then it sets those "accomplished" people up for failure as they eventually see their luck running out. They are very motivated to use less and less ethical tools for years to keep producing "results".
  - AlexandrB 3y ago
    This also describes the ultra-wealthy quite well, but few are in favor of a wealth tax.
    - seanp2k2 3y ago
      But because “some day it’ll be me and I won’t want to pay that”
    - pas 3y ago
      That doesn't make those few wrong. (Also the number of supporters seems to be growing. Piketty's book - Capital in the 21st century - had big waves, and it argues for a wealth tax, for example.)

DrScientist 3y ago

> well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Odd that tenure is singled out as the core of the problem. I'd say the undermining of tenure is a large part of the problem. Research is hard - very hard - you are being asked to discover stuff nobody else has - to push the boundaries of knowledge. I would argue that the idea that one failed research project, and you are out on your ear because of a gap in publishing - is a core part of the problem. In terms of medical doctors doing research - I'd argue that's a special dangerous mix - medical doctors need to have an element of self-belief just to be able to do their job ( life and death decisions ). Instilling it is part of the training. That self-belief, plus a lack ( in most cases ) of any in depth scientific research training is a dangerous mix. Medical training != science training. In this specific case I have no idea about the rights and wrongs.

pas 3y ago
Research might be hard, harder than whatever else the rest of the world is doing, but the tenure system is not giving us the best researchers. It rewards those that are at the right place at the right time, it's mostly luck of the current spectacle, and then it sets those "accomplished" people up for failure as they eventually see their luck running out. They are very motivated to use less and less ethical tools for years to keep producing "results".
- AlexandrB 3y ago
  This also describes the ultra-wealthy quite well, but few are in favor of a wealth tax.
  - seanp2k2 3y ago
    But because “some day it’ll be me and I won’t want to pay that”
  - pas 3y ago
    That doesn't make those few wrong. (Also the number of supporters seems to be growing. Piketty's book - Capital in the 21st century - had big waves, and it argues for a wealth tax, for example.)