6 ms·

As someone who did a PhD and published and read many papers (in ML) I believe vast majority of papers in ML are misleading. If in fact every paper that claims a

As someone who did a PhD and published and read many papers (in ML) I believe vast majority of papers in ML are misleading. If in fact every paper that claims a better performance over state of the art was true we would have solved AI by now. You see all sorts of problems when you dig deeper into the technical details of peer reviewed publications (even in top tier conferences) including misleading baselines, statistical insignificance of improvements, overfitting to test data, and in some cases just flat-out fabrication of results. I hope this is not as bad in medicine and health related research. But just thinking that some paper can be used against you in court to claim billions of dollars in damages makes me uneasy. Peer reviewed paper != science. Peer review is a crude filter on research that can both accept bad work and reject good work. There must be a higher bar for something that can be used in the court of law. Some sort of scientific consensus must be needed at least. It's easy to dismiss this because screw J&J. But I think we are all paying for these lawsuits through our insurances and taxes and higher drug prices. Not saying these lawsuits don't have merits, but I think there must be a higher bar for what is presented as evidence.

blitzar 3y ago
We need a Journal of Medicore Results, or a Journal of Tried it and it Didn't Work. As with social media and society in general, only the most inflated of headlines gets attention causing the gold rush of hyped up results. Nothing can be published without the hype and hyperbole, which then sets the benchmark for the next round of hype.
- xkcd1963 3y ago
  You say it as if that is something that should be accepted.
  - willis936 3y ago
    Is their desire to change it not evidence that they don't accept it? What they want is a real solution rather than to simply complain about the problem while doing nothing (which is closer to acceptance).
- fn-mote 3y ago
  > We need a Journal of Medicore Results, or a Journal of Tried it and it Didn't Work. Publication in this journal would be used as evidence against lawsuits. Cigarette companies paid for studies designed to produce no conclusive evidence. I don't see this as a route to progress. What if all you did wrong was follow the wrong process? The bacteria grows better at 10C than 30C. This journal would be full of results by fools and charlatans. The inclusion criteria would have to be much more complex for it to be useful.
  - Vaslo 3y ago
    Former chemist here. I disagree - you try many many reactions that fail. It would be good to just see what was tried so you can 1) see what was tried so you can change the conditions and try again or 2) avoid an approach altogether. I joke in many of the forums here about making The journal of Failed Chemistry. But I am very serious about saving time so 10 different PhD candidates don’t waste the same time I did.
    - thomastjeffery 3y ago
      Better yet, your documented "failure" may be an unknown path to "success" in someone else's research context. This would basically be the same as caching the results of a brute-force attack; except instead of trying to break the entropy of encryption, we are trying to unravel the entropy of chemistry and physics.
  - ryanmcbride 3y ago
    Bad actors will act bad in any environment in which they're able
- mplanchard 3y ago
  There are already a number of these: searching for “journal of negative results” turns up several
mike_hearn 3y ago
How do you measure consensus? There are many researchers who claim there's a consensus of whatever they happen to believe, but when counter-examples are pointed out they start No True Scotsmanning ("no real expert believes..."). Health related research is in a much worse state than ML unfortunately. ML suffers from metrics gaming and overfitting but there's probably not much outright fraud? Common estimates are that half of all medical papers are making false claims. This blog post of mine from a few years ago has some quotes from editors of well known medical journals where they express disbelief in their own published research base, and some examples of blatant fraud [0]. And this interview with Marc Andreessen indicates why VCs don't invest into biotech startups much [1]: I had a conversation with the long-time head of one of the big federal funding agencies for healthcare research who is also a very accomplished entrepreneur, and I said, “do you really think it’s true that 50-70% of biomedical research is fake?” This is a guy who has spent his life in this world. And he said “oh no, that’s not true at all. It’s 90%.” [Richard laughs]. I was like “holy shit,” I was flabbergasted that it could be 90%. [...] I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. In the same interview Andreessen references a study [2] that implies most medical/drug research stopped working overnight in the year 2000. The reason being that in that year the US govt started requiring drug trials to pre-register their hypothesis and then evaluate their results relative to that hypothesis i.e. they tried to prevent P-hacking. We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease [...] 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 [0] https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 [1] https://www.richardhanania.com/p/flying-x-wings-into-the-death-star https://www.richardhanania.com/p/flying-x-wings-into-the-dea... [2] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382&utm_content=buffer2135a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer https://journals.plos.org/plosone/article?id=10.1371/journal...
DrScientist 3y ago
When you become a world expert in a very narrow area of a field - like when you do a PhD - I found you discover that around 50% of papers are either pointless, misleading or wrong. Let me be clear - the largest proportion of those are the pointless ones ( the reader already knew what's in the paper ). Some of this is because research is hard, a lot of it is because of the immense pressure to publish and the strong bias for positive messages ( this approach is better, we discovered X ) to be published, rather than stuff that says - we tried this but it didn't work etc. > I hope this is not as bad in medicine and health related research. Same pressure, same human factors. There is of course an additional factor in cases like this - if your research hints at a link between talc and cancer - is it ethical not to publish while you wait another 5 years for a longer study?
- archo 3y ago
  'Publish or Perish' : https://en.wikipedia.org/wiki/Publish_or_perish https://en.wikipedia.org/wiki/Publish_or_perish
- mikrotikker 3y ago
  "Safe and effective"
spi 3y ago
Having a similar background (PhD in math, working in deep learning), so knowing nothing of the specifics of medicine, I think that, if anything, there the situation is probably _worse_. Results are less transparent, and you can't just read a paper and know if the results are made "look nicer", if you don't have the raw numbers of the experiments (which you almost never have). At least, in ML you typically can get an idea by reading the paper attentively, and if you're determinate enough, often you even have the source code to check. I guess that's why in medicine (and some other sciences), meta-reviews are so important: they read dozens or hundreds of papers on the same topic, compare methodologies, and try to deduce more realistic confidence intervals to limit statistical noise. That's how we now know that smoking definitely leads to higher probability of cancer, and most likely that's the case with red meat (and even more likely with smoked or otherwise preserved meat). The risk increases are small enough that a single study can never have the statistical power to prove it. Otherwise, when the famously wrong paper linking vaccines and autism came out, all families of poor autistic children could sue a lot (it was later proved that that one paper was fabricated with malice, but that doesn't change anything, if it were a random error the effect would be the same) This is why I share your concern for the billions of $ in sues due to this one paper. I have no idea who is abstractly "right" - it might be there was asbestos, or maybe not, and it might be that these talc products increased cancer likelihood, or maybe not. But health, and particularly cancer, is tricky: we all develop multiple cancers, sooner or later, unless we die before. Trying to pinpoint a specific cancer to a source is like fining the proverbial last straw that broke the camel's back - except that in this case it's not the sum of the straws that kill the camel, but rather one random straw across all its load. That said, I don't want to use this argument to wave company responsibilities away - of course companies who knowingly neglected safety measures, or endangered patient safety, should be punished with large fines (and possibly forced to shut down, in particularly extreme cases). As a European, I'm just uncomfortable with the idea of settling this in spectacular "patient vs. company" trials, where everything is dictated by the ability of lawyers and the will of judges, with results rather randomly ranging from nothing to literally making you rich (and again, your particular cancer might or might not be caused by that... and you can't really know, in any way). With the small additional caveat that you must have survived that in the first place, in order for the company to actually be significantly punished.
poolopolopolo 3y ago
There is saying (at least in spanish): "You are owner of your silence and slave of your words". Maybe you need your moderate your claiming before publishing something that could cause several damage to someone (That is why most news websites use potential words when publishing something: reported/may/thought/potentially/etc...).
euix 3y ago
I don't know if you looked into pursuing academic career and running the post-doc circuit (maybe that doesn't happen in ML) but academia is ultimately a business and academics are overwhelming pressured to publish and increase the stats.
pessimizer 3y ago
> It's easy to dismiss this because screw J&J. But I think we are all paying for these lawsuits through our insurances and taxes and higher drug prices. 1) You think these lawsuits have a bigger effect on insurance premiums than an increase in the incidence of ovarian cancer? 2) This is a choice we make as a society, and has little to do with lawsuits. J&J is not the victim of how we finance health care, it is the beneficiary.
- SideQuark 3y ago
  1) Did you run the numbers?
SkeuomorphicBee 3y ago
> I hope this is not as bad in medicine and health related research. I have friends and family working with medicine and health related research, and I don't think it is bad. In fact I'm a bit amazed at how good the Medicine academia is at science, and as comparison how bad computer-science (CSc) is at science. With very few exceptions, CSc papers don't have any actual science, they don't have a hypothesis and a method to test it, instead most papers in CSc can be summarized as "I did this new thing and I find it cool", the peers simply don't expect you to do actual science. In comparison, in Medicine papers you are expected to follow the scientific method to a T, they have a hypothesis and test it (often heavily relying on statistics), and yes some Medicine papers are bad (with bad methodology, bad sampling, or bad statistics like p-hacking) but even the bad ones try (and fail) to follow the scientific method (or at least pretend to try in case of malicious papers), because the standard is that high.
- greatfilter250 3y ago
  [dead]
thomastjeffery 3y ago
> If in fact every paper that claims a better performance over state of the art was true we would have solved AI by now. "Better" is not a single axis. By your logic, all I need to do is jump higher and higher, and sooner or later I will be able to fly! You aren't alone in this line of thinking. This fallacy has been written all over "AI" since we started calling it "AI"! That was the biggest mistake of all: by calling any arbitrary project "an Artificial Intelligence", we declare it an instance of the end goal! Now it just needs to be a "better", never "different". And that's what it means for a company to fail: it doesn't just need to be better, it needs to be different. If we don't allow failure to happen, we can never explore "different".
ouraf 3y ago
It is as bad in medicine, if not worse. Recently a very influential paper about alzheimer was pulled[1] because of almost everything you said, but for biology instead: data fabrication, overestimating numbers, etc. There's also a case related with cloning (which includes stem cell research) and massive fraud in South Korea 20 or so years ago.[2] It's a complicated matter: research is a beacon of light on top of a mountain of failed experiments. And not even the government will fund a mountain of failures if your shining discovery at the end can't bring returns to society (or shareholders) in greater value than what you took. And the pharmaceutical industry never settled for small results. And, unlike ML, it's much harder to test things as an outsider (you can hate the hype, but openAI, hugging face and all these "open" language models makes it much easier to learn, test, tweak and improve things without a master's degree in data engineering.). It's a grimy area to research even as a pastime. There's a lot of recorded practices both in research and production that makes you question how low can a person go for money 1: https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease https://www.science.org/content/article/potential-fabricatio... 2: https://youtu.be/ett_8wLJ87U https://youtu.be/ett_8wLJ87U

blitzar 3y ago

We need a Journal of Medicore Results, or a Journal of Tried it and it Didn't Work. As with social media and society in general, only the most inflated of headlines gets attention causing the gold rush of hyped up results. Nothing can be published without the hype and hyperbole, which then sets the benchmark for the next round of hype.

xkcd1963 3y ago
You say it as if that is something that should be accepted.
- willis936 3y ago
  Is their desire to change it not evidence that they don't accept it? What they want is a real solution rather than to simply complain about the problem while doing nothing (which is closer to acceptance).
fn-mote 3y ago
> We need a Journal of Medicore Results, or a Journal of Tried it and it Didn't Work. Publication in this journal would be used as evidence against lawsuits. Cigarette companies paid for studies designed to produce no conclusive evidence. I don't see this as a route to progress. What if all you did wrong was follow the wrong process? The bacteria grows better at 10C than 30C. This journal would be full of results by fools and charlatans. The inclusion criteria would have to be much more complex for it to be useful.
- Vaslo 3y ago
  Former chemist here. I disagree - you try many many reactions that fail. It would be good to just see what was tried so you can 1) see what was tried so you can change the conditions and try again or 2) avoid an approach altogether. I joke in many of the forums here about making The journal of Failed Chemistry. But I am very serious about saving time so 10 different PhD candidates don’t waste the same time I did.
  - thomastjeffery 3y ago
    Better yet, your documented "failure" may be an unknown path to "success" in someone else's research context. This would basically be the same as caching the results of a brute-force attack; except instead of trying to break the entropy of encryption, we are trying to unravel the entropy of chemistry and physics.
- ryanmcbride 3y ago
  Bad actors will act bad in any environment in which they're able
mplanchard 3y ago
There are already a number of these: searching for “journal of negative results” turns up several

mike_hearn 3y ago

How do you measure consensus? There are many researchers who claim there's a consensus of whatever they happen to believe, but when counter-examples are pointed out they start No True Scotsmanning ("no real expert believes..."). Health related research is in a much worse state than ML unfortunately. ML suffers from metrics gaming and overfitting but there's probably not much outright fraud? Common estimates are that half of all medical papers are making false claims. This blog post of mine from a few years ago has some quotes from editors of well known medical journals where they express disbelief in their own published research base, and some examples of blatant fraud [0]. And this interview with Marc Andreessen indicates why VCs don't invest into biotech startups much [1]: I had a conversation with the long-time head of one of the big federal funding agencies for healthcare research who is also a very accomplished entrepreneur, and I said, “do you really think it’s true that 50-70% of biomedical research is fake?” This is a guy who has spent his life in this world. And he said “oh no, that’s not true at all. It’s 90%.” [Richard laughs]. I was like “holy shit,” I was flabbergasted that it could be 90%. [...] I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. In the same interview Andreessen references a study [2] that implies most medical/drug research stopped working overnight in the year 2000. The reason being that in that year the US govt started requiring drug trials to pre-register their hypothesis and then evaluate their results relative to that hypothesis i.e. they tried to prevent P-hacking. We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease [...] 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 [0] https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 [1] https://www.richardhanania.com/p/flying-x-wings-into-the-death-star https://www.richardhanania.com/p/flying-x-wings-into-the-dea... [2] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382&utm_content=buffer2135a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer https://journals.plos.org/plosone/article?id=10.1371/journal...

DrScientist 3y ago

When you become a world expert in a very narrow area of a field - like when you do a PhD - I found you discover that around 50% of papers are either pointless, misleading or wrong. Let me be clear - the largest proportion of those are the pointless ones ( the reader already knew what's in the paper ). Some of this is because research is hard, a lot of it is because of the immense pressure to publish and the strong bias for positive messages ( this approach is better, we discovered X ) to be published, rather than stuff that says - we tried this but it didn't work etc. > I hope this is not as bad in medicine and health related research. Same pressure, same human factors. There is of course an additional factor in cases like this - if your research hints at a link between talc and cancer - is it ethical not to publish while you wait another 5 years for a longer study?

archo 3y ago
'Publish or Perish' : https://en.wikipedia.org/wiki/Publish_or_perish https://en.wikipedia.org/wiki/Publish_or_perish
mikrotikker 3y ago
"Safe and effective"

spi 3y ago

Having a similar background (PhD in math, working in deep learning), so knowing nothing of the specifics of medicine, I think that, if anything, there the situation is probably _worse_. Results are less transparent, and you can't just read a paper and know if the results are made "look nicer", if you don't have the raw numbers of the experiments (which you almost never have). At least, in ML you typically can get an idea by reading the paper attentively, and if you're determinate enough, often you even have the source code to check. I guess that's why in medicine (and some other sciences), meta-reviews are so important: they read dozens or hundreds of papers on the same topic, compare methodologies, and try to deduce more realistic confidence intervals to limit statistical noise. That's how we now know that smoking definitely leads to higher probability of cancer, and most likely that's the case with red meat (and even more likely with smoked or otherwise preserved meat). The risk increases are small enough that a single study can never have the statistical power to prove it. Otherwise, when the famously wrong paper linking vaccines and autism came out, all families of poor autistic children could sue a lot (it was later proved that that one paper was fabricated with malice, but that doesn't change anything, if it were a random error the effect would be the same) This is why I share your concern for the billions of $ in sues due to this one paper. I have no idea who is abstractly "right" - it might be there was asbestos, or maybe not, and it might be that these talc products increased cancer likelihood, or maybe not. But health, and particularly cancer, is tricky: we all develop multiple cancers, sooner or later, unless we die before. Trying to pinpoint a specific cancer to a source is like fining the proverbial last straw that broke the camel's back - except that in this case it's not the sum of the straws that kill the camel, but rather one random straw across all its load. That said, I don't want to use this argument to wave company responsibilities away - of course companies who knowingly neglected safety measures, or endangered patient safety, should be punished with large fines (and possibly forced to shut down, in particularly extreme cases). As a European, I'm just uncomfortable with the idea of settling this in spectacular "patient vs. company" trials, where everything is dictated by the ability of lawyers and the will of judges, with results rather randomly ranging from nothing to literally making you rich (and again, your particular cancer might or might not be caused by that... and you can't really know, in any way). With the small additional caveat that you must have survived that in the first place, in order for the company to actually be significantly punished.

poolopolopolo 3y ago

There is saying (at least in spanish): "You are owner of your silence and slave of your words". Maybe you need your moderate your claiming before publishing something that could cause several damage to someone (That is why most news websites use potential words when publishing something: reported/may/thought/potentially/etc...).

euix 3y ago

I don't know if you looked into pursuing academic career and running the post-doc circuit (maybe that doesn't happen in ML) but academia is ultimately a business and academics are overwhelming pressured to publish and increase the stats.

pessimizer 3y ago

> It's easy to dismiss this because screw J&J. But I think we are all paying for these lawsuits through our insurances and taxes and higher drug prices. 1) You think these lawsuits have a bigger effect on insurance premiums than an increase in the incidence of ovarian cancer? 2) This is a choice we make as a society, and has little to do with lawsuits. J&J is not the victim of how we finance health care, it is the beneficiary.

SideQuark 3y ago
1) Did you run the numbers?

SkeuomorphicBee 3y ago

> I hope this is not as bad in medicine and health related research. I have friends and family working with medicine and health related research, and I don't think it is bad. In fact I'm a bit amazed at how good the Medicine academia is at science, and as comparison how bad computer-science (CSc) is at science. With very few exceptions, CSc papers don't have any actual science, they don't have a hypothesis and a method to test it, instead most papers in CSc can be summarized as "I did this new thing and I find it cool", the peers simply don't expect you to do actual science. In comparison, in Medicine papers you are expected to follow the scientific method to a T, they have a hypothesis and test it (often heavily relying on statistics), and yes some Medicine papers are bad (with bad methodology, bad sampling, or bad statistics like p-hacking) but even the bad ones try (and fail) to follow the scientific method (or at least pretend to try in case of malicious papers), because the standard is that high.

greatfilter250 3y ago
[dead]

thomastjeffery 3y ago

> If in fact every paper that claims a better performance over state of the art was true we would have solved AI by now. "Better" is not a single axis. By your logic, all I need to do is jump higher and higher, and sooner or later I will be able to fly! You aren't alone in this line of thinking. This fallacy has been written all over "AI" since we started calling it "AI"! That was the biggest mistake of all: by calling any arbitrary project "an Artificial Intelligence", we declare it an instance of the end goal! Now it just needs to be a "better", never "different". And that's what it means for a company to fail: it doesn't just need to be better, it needs to be different. If we don't allow failure to happen, we can never explore "different".

ouraf 3y ago

It is as bad in medicine, if not worse. Recently a very influential paper about alzheimer was pulled[1] because of almost everything you said, but for biology instead: data fabrication, overestimating numbers, etc. There's also a case related with cloning (which includes stem cell research) and massive fraud in South Korea 20 or so years ago.[2] It's a complicated matter: research is a beacon of light on top of a mountain of failed experiments. And not even the government will fund a mountain of failures if your shining discovery at the end can't bring returns to society (or shareholders) in greater value than what you took. And the pharmaceutical industry never settled for small results. And, unlike ML, it's much harder to test things as an outsider (you can hate the hype, but openAI, hugging face and all these "open" language models makes it much easier to learn, test, tweak and improve things without a master's degree in data engineering.). It's a grimy area to research even as a pastime. There's a lot of recorded practices both in research and production that makes you question how low can a person go for money 1: https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease https://www.science.org/content/article/potential-fabricatio... 2: https://youtu.be/ett_8wLJ87U https://youtu.be/ett_8wLJ87U