5 ms·

I think one of the fundamental issues is that in science it is absolutely forbidden to accuse someone of fraud. If you do that you will be cancelled to use a mo

by 323 4y ago

I think one of the fundamental issues is that in science it is absolutely forbidden to accuse someone of fraud. If you do that you will be cancelled to use a modern term. And no peer reviewer dares question the integrity of the data or researcher, it's taken as gospel. So while many might know about, nobody dares say a thing.

sillysaurusx 4y ago
For what it’s worth, this hasn’t been my experience in ML. But I’ve had a limited window into academia. We were always questioning each other’s results. Especially when something seemed too good to be true. “Integrity” can take many forms, and it’s surprisingly easy to fool yourself when you’re doing work in the field. So the default assumption was that we were all fooling ourselves, not each other, unless proven otherwise.
- 323 4y ago
  But were you accusing others in public of intentional fraudulent activities? Because it's a different thing than "you have bad statistics here" or "wrong assumptions here".
  - sillysaurusx 4y ago
    That’s true.
- PeterisP 4y ago
  ML is a bit different because if the authors provide sufficient data, then any fraud is trivially obvious on replication and if they don't provide sufficient data, then that is a reasonable criticism on its own without having to go into motives. However, in other areas where you have "real world" experiments, you don't even expect the experiments to replicate - two clinical trials on different sets of patients won't necessarily yield the same results, and different results when repeating a biological experiment does not necessarily imply fraud; we know that in this domain (unlike ML) we sometimes do have unknown confounders that experiments don't control for.
- caycep 4y ago
  we have peer review....but peer review is human, and some fields, it's a small world. I'm not in that particular field of neuroscience (albeit Alzheimer's patients do come in to the clinic...because where else are they going to go?). But when I was in the research world, a big factor was who you knew, and who you could cozy up to, in terms of getting grants, papers published in top journals, etc. Who you ate lunch w/ in big conferences. That being said, fortunately, there was an element of recognition of skill/ability and good work, as well as a decent amount of (sometimes irrelevant) challenge/response and general peacock-ing in terms of the peer review...but it was a reminder that science is above all, a human endeavor and not immune to humanity's sins.
kensai 4y ago
You will not be cancelled as long as you can back your claim. Like in everything in this world. The equivalence of cancelling in HN is of course downvoting to oblivion. We have that tool here as well. Controversial opinions are heard as long as they have backing or/and something important to add to the conversation.
- photochemsyn 4y ago
  You can report fraud to the ORI (Office of Research Integrity) and they won't do a thing about it. They won't open an independent audit of any federally-funded academic researcher until after the fraud has been publicly exposed in the media, as this one has been. Even then: > "The agency’s reply, which Schrag shared with Science, noted that complaints deemed credible will go to the Department of Health and Human Services Office of Research Integrity (ORI) for review. That agency could then instruct grantee universities to investigate prior to a final ORI review, a process that can take years and remains confidential absent an official misconduct finding. To Science, NIH said it takes research misconduct seriously, but otherwise declined to comment." https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease https://www.science.org/content/article/potential-fabricatio... There is simply no willingness on the part of federal funding agencies to investigate their own grantees after they receive credible evidence of fraud.
bsder 4y ago
> So while many might know about, nobody dares say a thing. Maybe, but it's more likely that compiling solid evidence for fraud is just far too resource intensive and there is no reward for replication--especially negative replication. And committing the fraud is far too tempting--you got your needed publication for tenure, and you're mostly safe as long as nobody accidentally makes your paper the cornerstone for something that becomes famous. As for peers, you can spend your time moving your own ideas forward, or you can take a detour to prove one particular important researcher's ideas wrong. It's pretty clear which is going to be more beneficial to your career. As the article points out, this was triggered by some short sellers looking to make a buck--not anybody in the field itself.

by 323 4y ago

sillysaurusx 4y ago

For what it’s worth, this hasn’t been my experience in ML. But I’ve had a limited window into academia. We were always questioning each other’s results. Especially when something seemed too good to be true. “Integrity” can take many forms, and it’s surprisingly easy to fool yourself when you’re doing work in the field. So the default assumption was that we were all fooling ourselves, not each other, unless proven otherwise.

323 4y ago
But were you accusing others in public of intentional fraudulent activities? Because it's a different thing than "you have bad statistics here" or "wrong assumptions here".
- sillysaurusx 4y ago
  That’s true.
PeterisP 4y ago
ML is a bit different because if the authors provide sufficient data, then any fraud is trivially obvious on replication and if they don't provide sufficient data, then that is a reasonable criticism on its own without having to go into motives. However, in other areas where you have "real world" experiments, you don't even expect the experiments to replicate - two clinical trials on different sets of patients won't necessarily yield the same results, and different results when repeating a biological experiment does not necessarily imply fraud; we know that in this domain (unlike ML) we sometimes do have unknown confounders that experiments don't control for.
caycep 4y ago
we have peer review....but peer review is human, and some fields, it's a small world. I'm not in that particular field of neuroscience (albeit Alzheimer's patients do come in to the clinic...because where else are they going to go?). But when I was in the research world, a big factor was who you knew, and who you could cozy up to, in terms of getting grants, papers published in top journals, etc. Who you ate lunch w/ in big conferences. That being said, fortunately, there was an element of recognition of skill/ability and good work, as well as a decent amount of (sometimes irrelevant) challenge/response and general peacock-ing in terms of the peer review...but it was a reminder that science is above all, a human endeavor and not immune to humanity's sins.

kensai 4y ago

You will not be cancelled as long as you can back your claim. Like in everything in this world. The equivalence of cancelling in HN is of course downvoting to oblivion. We have that tool here as well. Controversial opinions are heard as long as they have backing or/and something important to add to the conversation.

photochemsyn 4y ago
You can report fraud to the ORI (Office of Research Integrity) and they won't do a thing about it. They won't open an independent audit of any federally-funded academic researcher until after the fraud has been publicly exposed in the media, as this one has been. Even then: > "The agency’s reply, which Schrag shared with Science, noted that complaints deemed credible will go to the Department of Health and Human Services Office of Research Integrity (ORI) for review. That agency could then instruct grantee universities to investigate prior to a final ORI review, a process that can take years and remains confidential absent an official misconduct finding. To Science, NIH said it takes research misconduct seriously, but otherwise declined to comment." https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease https://www.science.org/content/article/potential-fabricatio... There is simply no willingness on the part of federal funding agencies to investigate their own grantees after they receive credible evidence of fraud.

bsder 4y ago

> So while many might know about, nobody dares say a thing. Maybe, but it's more likely that compiling solid evidence for fraud is just far too resource intensive and there is no reward for replication--especially negative replication. And committing the fraud is far too tempting--you got your needed publication for tenure, and you're mostly safe as long as nobody accidentally makes your paper the cornerstone for something that becomes famous. As for peers, you can spend your time moving your own ideas forward, or you can take a detour to prove one particular important researcher's ideas wrong. It's pretty clear which is going to be more beneficial to your career. As the article points out, this was triggered by some short sellers looking to make a buck--not anybody in the field itself.