7 ms·

For what it’s worth, this hasn’t been my experience in ML. But I’ve had a limited window into academia. We were always questioning each other’s results. Especi

For what it’s worth, this hasn’t been my experience in ML. But I’ve had a limited window into academia. We were always questioning each other’s results. Especially when something seemed too good to be true. “Integrity” can take many forms, and it’s surprisingly easy to fool yourself when you’re doing work in the field. So the default assumption was that we were all fooling ourselves, not each other, unless proven otherwise.

323 4y ago
But were you accusing others in public of intentional fraudulent activities? Because it's a different thing than "you have bad statistics here" or "wrong assumptions here".
- sillysaurusx 4y ago
  That’s true.
PeterisP 4y ago
ML is a bit different because if the authors provide sufficient data, then any fraud is trivially obvious on replication and if they don't provide sufficient data, then that is a reasonable criticism on its own without having to go into motives. However, in other areas where you have "real world" experiments, you don't even expect the experiments to replicate - two clinical trials on different sets of patients won't necessarily yield the same results, and different results when repeating a biological experiment does not necessarily imply fraud; we know that in this domain (unlike ML) we sometimes do have unknown confounders that experiments don't control for.
caycep 4y ago
we have peer review....but peer review is human, and some fields, it's a small world. I'm not in that particular field of neuroscience (albeit Alzheimer's patients do come in to the clinic...because where else are they going to go?). But when I was in the research world, a big factor was who you knew, and who you could cozy up to, in terms of getting grants, papers published in top journals, etc. Who you ate lunch w/ in big conferences. That being said, fortunately, there was an element of recognition of skill/ability and good work, as well as a decent amount of (sometimes irrelevant) challenge/response and general peacock-ing in terms of the peer review...but it was a reminder that science is above all, a human endeavor and not immune to humanity's sins.

323 4y ago

But were you accusing others in public of intentional fraudulent activities? Because it's a different thing than "you have bad statistics here" or "wrong assumptions here".

sillysaurusx 4y ago
That’s true.

PeterisP 4y ago

ML is a bit different because if the authors provide sufficient data, then any fraud is trivially obvious on replication and if they don't provide sufficient data, then that is a reasonable criticism on its own without having to go into motives. However, in other areas where you have "real world" experiments, you don't even expect the experiments to replicate - two clinical trials on different sets of patients won't necessarily yield the same results, and different results when repeating a biological experiment does not necessarily imply fraud; we know that in this domain (unlike ML) we sometimes do have unknown confounders that experiments don't control for.

caycep 4y ago

we have peer review....but peer review is human, and some fields, it's a small world. I'm not in that particular field of neuroscience (albeit Alzheimer's patients do come in to the clinic...because where else are they going to go?). But when I was in the research world, a big factor was who you knew, and who you could cozy up to, in terms of getting grants, papers published in top journals, etc. Who you ate lunch w/ in big conferences. That being said, fortunately, there was an element of recognition of skill/ability and good work, as well as a decent amount of (sometimes irrelevant) challenge/response and general peacock-ing in terms of the peer review...but it was a reminder that science is above all, a human endeavor and not immune to humanity's sins.