5 ms·

I have a strong feeling that, if people really put an effort into reading and replicating more papers, we would find that a lot of what's being published is sim

I have a strong feeling that, if people really put an effort into reading and replicating more papers, we would find that a lot of what's being published is simply meaningless. In grad school I had a subletting roommate for a while who was writing code to match some experimental data with a model. He showed me his model. It was quite literally making random combinations of various trigonometric functions, absolute value, logarithms, polynomials, exponents, etc. into equations that were like a whole page long and just wiggling them around. He was convinced that he was on a path to a revolution in understanding the functional form of his (biological) data, and I believe his research PI was onboard. I guess "overfitted" never made it into the curriculum.

gameswithgo 6y ago
isn’t the saying, 80% of everything is garbage?
- bryanrasmussen 6y ago
  Sturgeon's Law https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law "ninety percent of everything is crap."
- cproctor 6y ago
  We better work on making more stuff.
  - omgwtfbyobbq 6y ago
    Better stuff?
    - oneplane 6y ago
      If you have more stuff, and 10% is good, than that stuff can be more of the better.
- goatlover 6y ago
  Yeah, but one would hope that science has a higher standard. 80% garbage results in science sounds catastrophic to our understanding of the world, and in particular when it comes to making policies based on that science.
  - behringer 6y ago
    You do realize we've been at this for a few thousand years, right? Catastrophic is putting it mildly.
  - adamsea 6y ago
    There's the saying "science advances one funeral at at time." '‘A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.’ This principle was famously laid out by German theoretical physicist Max Planck in 1950 and it turns out that he was right, according to a new study.' https://www.chemistryworld.com/news/science-really-does-advance-one-funeral-at-a-time-study-suggests/3010961.article https://www.chemistryworld.com/news/science-really-does-adva... Also the story of Ignaz Semmelweis who discovered that if doctors washed their hands it reduced deaths during childbirth - but for a variety of reasons his findings were resisted. https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... Point being, as awesome as science is, it's still a human enterprise, and humans are still, well, human.
  - barrenko 6y ago
    Unfortunately - like bs, power law is fractal.
fock 6y ago
I work in computational materials science (where ML brings funding) and a funny paper of this kind is here: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.114.105503 https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.11... - they are literally trying out 100000s of possible combinations by brute force, to build a "physical model". Then they go on conferences and brag about it, because they have to (otr they know it's bs). Datasets are soso (you can have a look at QM9...) and for more specialized things, people generally don't bother trying to benchmark or compare their results on a common reference. It's just something new... And with all that: even without doing fancy statistical methods without knowing too much about it, your theoretical computations might not make so much sense (at least in the sheer number which is pumped out and published)...
- jordigh 6y ago
  > (otr they know it's bs). Well, that's a new acronym for me. I wonder where it came from. Apparently it's "on the real". Sounds like AAVE?
  - EsotericAlgo 6y ago
    I thought it was off the record like the old Pidgin plugin or https://en.wikipedia.org/wiki/Off-the-Record_Messaging https://en.wikipedia.org/wiki/Off-the-Record_Messaging.
  - ortusdux 6y ago
    typo for 'or'?
  - nhooyr 6y ago
    It means off the record.
  - fock 6y ago
    off-the-record (not written, not cited, but available in personal discussions).
    - 6y ago
throwaway_pdp09 6y ago
Not my area at all but isn't this genetic algorithm type stuff?
zozbot234 6y ago
> It was quite literally making random combinations of various trigonometric functions, absolute value, logarithms, polynomials, exponents, etc. into equations that were like a whole page long and just wiggling them around. Technically, we call that a "neural network". Or "AI".
- aaaxyz 6y ago
  It reminds me more of a genetic algorithm
  - damagednoob 6y ago
    Everytime I think I'm getting somewhere with a GA I eventually realise I've just created a guided monte carlo simulation.
- mjburgess 6y ago
  This answers the question "will we ever have AI as smart as a human?" Yes. It just turns out it's a particular human, whose analysis is very very dumb.
hannob 6y ago
> I have a strong feeling that, if people really put an effort into reading and replicating more papers, we would find that a lot of what's being published is simply meaningless. People have figured that out long ago [1] (I know the author of that paper lately turned somewhat controversial, but that doesn't change his findings). It's not very widely known in the general public. But if you understand some basic issues like p-hacking and publication bias and combine that with the knowledge that most scientific fields don't do anything about these issues, there can hardly be any doubt that a lot of research is rubbish. [1] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

gameswithgo 6y ago

isn’t the saying, 80% of everything is garbage?

bryanrasmussen 6y ago
Sturgeon's Law https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law "ninety percent of everything is crap."
cproctor 6y ago
We better work on making more stuff.
- omgwtfbyobbq 6y ago
  Better stuff?
  - oneplane 6y ago
    If you have more stuff, and 10% is good, than that stuff can be more of the better.
goatlover 6y ago
Yeah, but one would hope that science has a higher standard. 80% garbage results in science sounds catastrophic to our understanding of the world, and in particular when it comes to making policies based on that science.
- behringer 6y ago
  You do realize we've been at this for a few thousand years, right? Catastrophic is putting it mildly.
- adamsea 6y ago
  There's the saying "science advances one funeral at at time." '‘A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.’ This principle was famously laid out by German theoretical physicist Max Planck in 1950 and it turns out that he was right, according to a new study.' https://www.chemistryworld.com/news/science-really-does-advance-one-funeral-at-a-time-study-suggests/3010961.article https://www.chemistryworld.com/news/science-really-does-adva... Also the story of Ignaz Semmelweis who discovered that if doctors washed their hands it reduced deaths during childbirth - but for a variety of reasons his findings were resisted. https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... Point being, as awesome as science is, it's still a human enterprise, and humans are still, well, human.
- barrenko 6y ago
  Unfortunately - like bs, power law is fractal.

fock 6y ago

I work in computational materials science (where ML brings funding) and a funny paper of this kind is here: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.114.105503 https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.11... - they are literally trying out 100000s of possible combinations by brute force, to build a "physical model". Then they go on conferences and brag about it, because they have to (otr they know it's bs). Datasets are soso (you can have a look at QM9...) and for more specialized things, people generally don't bother trying to benchmark or compare their results on a common reference. It's just something new... And with all that: even without doing fancy statistical methods without knowing too much about it, your theoretical computations might not make so much sense (at least in the sheer number which is pumped out and published)...

jordigh 6y ago
> (otr they know it's bs). Well, that's a new acronym for me. I wonder where it came from. Apparently it's "on the real". Sounds like AAVE?
- EsotericAlgo 6y ago
  I thought it was off the record like the old Pidgin plugin or https://en.wikipedia.org/wiki/Off-the-Record_Messaging https://en.wikipedia.org/wiki/Off-the-Record_Messaging.
- ortusdux 6y ago
  typo for 'or'?
- nhooyr 6y ago
  It means off the record.
- fock 6y ago
  off-the-record (not written, not cited, but available in personal discussions).
  - 6y ago

throwaway_pdp09 6y ago

Not my area at all but isn't this genetic algorithm type stuff?

zozbot234 6y ago

> It was quite literally making random combinations of various trigonometric functions, absolute value, logarithms, polynomials, exponents, etc. into equations that were like a whole page long and just wiggling them around. Technically, we call that a "neural network". Or "AI".

aaaxyz 6y ago
It reminds me more of a genetic algorithm
- damagednoob 6y ago
  Everytime I think I'm getting somewhere with a GA I eventually realise I've just created a guided monte carlo simulation.
mjburgess 6y ago
This answers the question "will we ever have AI as smart as a human?" Yes. It just turns out it's a particular human, whose analysis is very very dumb.

hannob 6y ago

> I have a strong feeling that, if people really put an effort into reading and replicating more papers, we would find that a lot of what's being published is simply meaningless. People have figured that out long ago [1] (I know the author of that paper lately turned somewhat controversial, but that doesn't change his findings). It's not very widely known in the general public. But if you understand some basic issues like p-hacking and publication bias and combine that with the knowledge that most scientific fields don't do anything about these issues, there can hardly be any doubt that a lot of research is rubbish. [1] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...