6 ms·

as a non-scientist outsider, a few questions if you don't mind: 1. is it common for a non-reproduced experiment to gain prominence as a leading hypothesis and

as a non-scientist outsider, a few questions if you don't mind: 1. is it common for a non-reproduced experiment to gain prominence as a leading hypothesis and spur related research? 2. if so, why? if not, why did this hypothesis take hold? 3. how can we make science more reproducible? reproducibility is held as a tenet of science, but this example, assuming it is true, would violate the principle.

londons_explore 4y ago
Trust, but don't verify, because verification is expensive and unrewarding both financially and career-wise
ethbr0 4y ago
1. Yes. Science has a snowball effect as a result of committee-driven grant decisions. Research in a hot current topic attracts more grant funding more reliably. 2. It does because some fields of science are intrinsically long-duration, highly sensitive to variables, or only testable at scales that exceed our experimental capability. That's just a consequence of physics and natural laws of reality. E.g. nutrition, chemical toxicity, economics, high energy physics. 3. We can cheat and find proxies or more tenable micro-systems to experiment on. But often those have their own problems (e.g. rodent models) or aren't feasible. Reproducibility is a goal of science. It's not always an achievable goal. When it's not, we do the best we can, as with drug testing pipelines.
- panabee 4y ago
  thanks for this response. given the prevalence of non-reproducibility, what other fields do you believe have suspect leading hypotheses?
  - Quekid5 4y ago
    Not the person you replied to, but... Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks. Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc. Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science. [0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).
  - Blahah 4y ago
    In general, anything that's at the bleeding edge of theoretical understanding and intersects with commercial interest or draws social/media hype has a relatively low probability of standing the test of time. Whether that's replicability, applicability, or just validity. The most exciting hypotheses tend to go one of two ways. They are rapidly supported with independent evidence, by being applied or replicated. Or they draw a lot of attention and money which helps them persist in spite of, or in the absence of, evidence. Charlatans, purveyors of snake oil, and (most often) people who for whatever reason don't want to be seen to be wrong - they exist in every walk of life. Science is the same. The incentives in the system strongly select for these people in the 'leading hypotheses' space.
  - mike_hearn 4y ago
    Here's a couple: 1. Twitter bot / misinformation research. A surprisingly large field in which papers are almost never replicable because they don't supply the actual tweets, but only opaque IDs and classifications. Trying to cross-check them is futile because by the time you tried months later many of the accounts have been suspended. They could easily supply the contents of the tweets they scraped along with account metadata, but don't. The few times I did deeper checks of these papers I always found some accounts that were identified as bots but weren't suspended, and on manual inspection were very obviously human. This field also has the problem of being increasingly based on ML pseudo-science. 2. Germ theory! It can't explain several aspects of the epidemiology of respiratory diseases. It's probably not wrong but is certainly incomplete. As we saw with COVID, models based on simple germ theory always make wrong predictions, but this problem pre-dates COVID. It was known for a long time that standard germ theory fails to explain the behavior of influenza, for example, why it's seasonal, why waves peak and enter decline before everyone is infected, why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. The issues here are deeper than reproducibility though. Even if bot papers were reproducible they would still be wrong because their methodologies are invalid and cannot support the stated conclusions. It's important not to get too focused on mere replication.
    - CRConrad 4y ago
      > 2. Germ theory! It can't explain [...] the behavior of influenza, for example, why it's seasonal, Huh? I thought that's so well-known (must be, if even I have heard of it) that it goes without saying: Spring-summer, people go out and breathe fresh air a bit apart; autumn-winter, everyone goes inside and coughs their germs at each other. > why waves peak and enter decline before everyone is infected, The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable. (Pretty much inevitable, really, isn't it?) > why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, See first point above. > why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. Someone always flies in supplies... I mean, WTF -- "Germ theory"? Isn't that what it was called in the 19th century, when there still was any doubt about it? What next; "Gravity is just a theory!"?
- CRConrad 4y ago
  > Reproducibility is a goal of science. It's not always an achievable goal. And here I thought the goal of science was to find the truth[1] about how the world works, with reproducibility as one of the means to do that. ___ [1]: Or at least get as close as possible, hopefully asymptotically ever closer, to the truth.
ta988 4y ago
One of the issues is that by chance the models used have an unexpected behavior that seems to confirm the initial hypothesis is sound. And this is not rare. Sometimes it is fraudulent, people know things are faulty but continue anyway. In other cases it could be by ignorance of the existence of the phenomenon behind the faulty behavior or simply because that phenomenon has not been discovered yet.
elihu 4y ago
The standard of truth in science is reproducibility. But publication usually happens before results have been independently reproduced. Papers are peer reviewed, but generally peer reviewers are just ensuring that the results are sufficiently interesting to warrant publication and there aren't any obvious methodology mistakes. For the purpose of funding, publication in important journals or conference proceedings is the important thing. And the public generally tends to treat publication as the standard of truth. But publication kind of works on the honor system and is susceptible to fabricated results. Maybe there should be more funding towards reproducing (or not) important results?
marsven_422 4y ago

londons_explore 4y ago

Trust, but don't verify, because verification is expensive and unrewarding both financially and career-wise

ethbr0 4y ago

1. Yes. Science has a snowball effect as a result of committee-driven grant decisions. Research in a hot current topic attracts more grant funding more reliably. 2. It does because some fields of science are intrinsically long-duration, highly sensitive to variables, or only testable at scales that exceed our experimental capability. That's just a consequence of physics and natural laws of reality. E.g. nutrition, chemical toxicity, economics, high energy physics. 3. We can cheat and find proxies or more tenable micro-systems to experiment on. But often those have their own problems (e.g. rodent models) or aren't feasible. Reproducibility is a goal of science. It's not always an achievable goal. When it's not, we do the best we can, as with drug testing pipelines.

panabee 4y ago
thanks for this response. given the prevalence of non-reproducibility, what other fields do you believe have suspect leading hypotheses?
- Quekid5 4y ago
  Not the person you replied to, but... Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks. Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc. Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science. [0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).
- Blahah 4y ago
  In general, anything that's at the bleeding edge of theoretical understanding and intersects with commercial interest or draws social/media hype has a relatively low probability of standing the test of time. Whether that's replicability, applicability, or just validity. The most exciting hypotheses tend to go one of two ways. They are rapidly supported with independent evidence, by being applied or replicated. Or they draw a lot of attention and money which helps them persist in spite of, or in the absence of, evidence. Charlatans, purveyors of snake oil, and (most often) people who for whatever reason don't want to be seen to be wrong - they exist in every walk of life. Science is the same. The incentives in the system strongly select for these people in the 'leading hypotheses' space.
- mike_hearn 4y ago
  Here's a couple: 1. Twitter bot / misinformation research. A surprisingly large field in which papers are almost never replicable because they don't supply the actual tweets, but only opaque IDs and classifications. Trying to cross-check them is futile because by the time you tried months later many of the accounts have been suspended. They could easily supply the contents of the tweets they scraped along with account metadata, but don't. The few times I did deeper checks of these papers I always found some accounts that were identified as bots but weren't suspended, and on manual inspection were very obviously human. This field also has the problem of being increasingly based on ML pseudo-science. 2. Germ theory! It can't explain several aspects of the epidemiology of respiratory diseases. It's probably not wrong but is certainly incomplete. As we saw with COVID, models based on simple germ theory always make wrong predictions, but this problem pre-dates COVID. It was known for a long time that standard germ theory fails to explain the behavior of influenza, for example, why it's seasonal, why waves peak and enter decline before everyone is infected, why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. The issues here are deeper than reproducibility though. Even if bot papers were reproducible they would still be wrong because their methodologies are invalid and cannot support the stated conclusions. It's important not to get too focused on mere replication.
  - CRConrad 4y ago
    > 2. Germ theory! It can't explain [...] the behavior of influenza, for example, why it's seasonal, Huh? I thought that's so well-known (must be, if even I have heard of it) that it goes without saying: Spring-summer, people go out and breathe fresh air a bit apart; autumn-winter, everyone goes inside and coughs their germs at each other. > why waves peak and enter decline before everyone is infected, The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable. (Pretty much inevitable, really, isn't it?) > why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, See first point above. > why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. Someone always flies in supplies... I mean, WTF -- "Germ theory"? Isn't that what it was called in the 19th century, when there still was any doubt about it? What next; "Gravity is just a theory!"?
CRConrad 4y ago
> Reproducibility is a goal of science. It's not always an achievable goal. And here I thought the goal of science was to find the truth[1] about how the world works, with reproducibility as one of the means to do that. ___ [1]: Or at least get as close as possible, hopefully asymptotically ever closer, to the truth.

ta988 4y ago

One of the issues is that by chance the models used have an unexpected behavior that seems to confirm the initial hypothesis is sound. And this is not rare. Sometimes it is fraudulent, people know things are faulty but continue anyway. In other cases it could be by ignorance of the existence of the phenomenon behind the faulty behavior or simply because that phenomenon has not been discovered yet.

elihu 4y ago

The standard of truth in science is reproducibility. But publication usually happens before results have been independently reproduced. Papers are peer reviewed, but generally peer reviewers are just ensuring that the results are sufficiently interesting to warrant publication and there aren't any obvious methodology mistakes. For the purpose of funding, publication in important journals or conference proceedings is the important thing. And the public generally tends to treat publication as the standard of truth. But publication kind of works on the honor system and is susceptible to fabricated results. Maybe there should be more funding towards reproducing (or not) important results?

marsven_422 4y ago