4 ms·

thanks for this response. given the prevalence of non-reproducibility, what other fields do you believe have suspect leading hypotheses?

Quekid5 4y ago
Not the person you replied to, but... Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks. Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc. Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science. [0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).
Blahah 4y ago
In general, anything that's at the bleeding edge of theoretical understanding and intersects with commercial interest or draws social/media hype has a relatively low probability of standing the test of time. Whether that's replicability, applicability, or just validity. The most exciting hypotheses tend to go one of two ways. They are rapidly supported with independent evidence, by being applied or replicated. Or they draw a lot of attention and money which helps them persist in spite of, or in the absence of, evidence. Charlatans, purveyors of snake oil, and (most often) people who for whatever reason don't want to be seen to be wrong - they exist in every walk of life. Science is the same. The incentives in the system strongly select for these people in the 'leading hypotheses' space.
mike_hearn 4y ago
Here's a couple: 1. Twitter bot / misinformation research. A surprisingly large field in which papers are almost never replicable because they don't supply the actual tweets, but only opaque IDs and classifications. Trying to cross-check them is futile because by the time you tried months later many of the accounts have been suspended. They could easily supply the contents of the tweets they scraped along with account metadata, but don't. The few times I did deeper checks of these papers I always found some accounts that were identified as bots but weren't suspended, and on manual inspection were very obviously human. This field also has the problem of being increasingly based on ML pseudo-science. 2. Germ theory! It can't explain several aspects of the epidemiology of respiratory diseases. It's probably not wrong but is certainly incomplete. As we saw with COVID, models based on simple germ theory always make wrong predictions, but this problem pre-dates COVID. It was known for a long time that standard germ theory fails to explain the behavior of influenza, for example, why it's seasonal, why waves peak and enter decline before everyone is infected, why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. The issues here are deeper than reproducibility though. Even if bot papers were reproducible they would still be wrong because their methodologies are invalid and cannot support the stated conclusions. It's important not to get too focused on mere replication.
- CRConrad 4y ago
  > 2. Germ theory! It can't explain [...] the behavior of influenza, for example, why it's seasonal, Huh? I thought that's so well-known (must be, if even I have heard of it) that it goes without saying: Spring-summer, people go out and breathe fresh air a bit apart; autumn-winter, everyone goes inside and coughs their germs at each other. > why waves peak and enter decline before everyone is infected, The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable. (Pretty much inevitable, really, isn't it?) > why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, See first point above. > why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. Someone always flies in supplies... I mean, WTF -- "Germ theory"? Isn't that what it was called in the 19th century, when there still was any doubt about it? What next; "Gravity is just a theory!"?
  - mike_hearn 4y ago
    That's a popular ad-hoc "street" theory (it's easy to come up with those) but formally speaking that idea has never been tested or proven; it's not a part of germ theory. If you look at epidemiological models, they rarely model seasonality for that reason. The models used for COVID for example, didn't have any notion of seasonality in them for a long time and many still don't, because what you just said isn't actually scientifically accepted. And it would probably run into some practical problems if you tried to nail it down, like the fact that for many people they are office or factory workers and the amount of time they spend outside doesn't change all that radically during summer, certainly not enough to make the difference between explosive spread and total eradication. "The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable" That's why the wave slows down yes, but again, this ad-hoc notion doesn't work. Try it out on paper or code up a quick epi model yourself and then compare against real world case data. The waves always end long before predicted, even when there are still tons of uninfected people available to infect and people wandering around who should easily infect them. This is one of the reasons why COVID modelling failed so badly. They coded up the exact simple germ theory model you're describing here, and of course it yields a single giant wave whereas what is seen in reality is a long series of small waves that start and end in ways that aren't predicted by the theory. "See first point above." See my reply. I can assure you, that the epidemiology of influenza really isn't as simple as "people going outside in summer". Take a look at some of the research papers from the 80s exploring this topic. There are still plenty of people interacting with each other closely indoors even in summer months, yet flu completely disappears. It's not something that ties in any obvious way back to accepted theory. "Someone always flies in supplies..." That's not what "totally isolated" means. In the cases in question there was no contact with the outside world whatsoever. In one case, at a British polar base, they mounted a very thorough investigation after an outbreak of cold virus after 17 weeks of total isolation. They checked if any new supply crates had been opened, etc, but no. They couldn't identify anywhere that new viruses might have been introduced to the base. https://www.cambridge.org/core/journals/epidemiology-and-infection/article/an-outbreak-of-common-colds-at-an-antarctic-base-after-seventeen-weeks-of-complete-isolation/1D3A49463583D06CEACE1CCF9C1A25B4 https://www.cambridge.org/core/journals/epidemiology-and-inf... One of the more popular sub-theories (though largely ignored by epidemiologists) to try and explain these things is the possibility that many people are continuously infected at a sub-symptomatic level, that the immune system never 100% wipes out the viral infection in these people, just keeps it in check. Then something happens to slightly knock the immune system out of balance for a moment, like a sudden change in temperature, and the infection is able to re-gain a foothold and starts replicating out of control again. So that'd be why people just show up everywhere at once spontaneously infected without any obvious index cases.

Quekid5 4y ago

Not the person you replied to, but... Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks. Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc. Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science. [0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).

Blahah 4y ago

In general, anything that's at the bleeding edge of theoretical understanding and intersects with commercial interest or draws social/media hype has a relatively low probability of standing the test of time. Whether that's replicability, applicability, or just validity. The most exciting hypotheses tend to go one of two ways. They are rapidly supported with independent evidence, by being applied or replicated. Or they draw a lot of attention and money which helps them persist in spite of, or in the absence of, evidence. Charlatans, purveyors of snake oil, and (most often) people who for whatever reason don't want to be seen to be wrong - they exist in every walk of life. Science is the same. The incentives in the system strongly select for these people in the 'leading hypotheses' space.

mike_hearn 4y ago

Here's a couple: 1. Twitter bot / misinformation research. A surprisingly large field in which papers are almost never replicable because they don't supply the actual tweets, but only opaque IDs and classifications. Trying to cross-check them is futile because by the time you tried months later many of the accounts have been suspended. They could easily supply the contents of the tweets they scraped along with account metadata, but don't. The few times I did deeper checks of these papers I always found some accounts that were identified as bots but weren't suspended, and on manual inspection were very obviously human. This field also has the problem of being increasingly based on ML pseudo-science. 2. Germ theory! It can't explain several aspects of the epidemiology of respiratory diseases. It's probably not wrong but is certainly incomplete. As we saw with COVID, models based on simple germ theory always make wrong predictions, but this problem pre-dates COVID. It was known for a long time that standard germ theory fails to explain the behavior of influenza, for example, why it's seasonal, why waves peak and enter decline before everyone is infected, why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. The issues here are deeper than reproducibility though. Even if bot papers were reproducible they would still be wrong because their methodologies are invalid and cannot support the stated conclusions. It's important not to get too focused on mere replication.

CRConrad 4y ago
> 2. Germ theory! It can't explain [...] the behavior of influenza, for example, why it's seasonal, Huh? I thought that's so well-known (must be, if even I have heard of it) that it goes without saying: Spring-summer, people go out and breathe fresh air a bit apart; autumn-winter, everyone goes inside and coughs their germs at each other. > why waves peak and enter decline before everyone is infected, The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable. (Pretty much inevitable, really, isn't it?) > why variants disappear totally instead of coexisting, why flu season seems to start everywhere in season almost simultaneously, See first point above. > why there have been outbreaks of respiratory viruses in totally isolated environments like arctic bases, and so on. Someone always flies in supplies... I mean, WTF -- "Germ theory"? Isn't that what it was called in the 19th century, when there still was any doubt about it? What next; "Gravity is just a theory!"?
- mike_hearn 4y ago
  That's a popular ad-hoc "street" theory (it's easy to come up with those) but formally speaking that idea has never been tested or proven; it's not a part of germ theory. If you look at epidemiological models, they rarely model seasonality for that reason. The models used for COVID for example, didn't have any notion of seasonality in them for a long time and many still don't, because what you just said isn't actually scientifically accepted. And it would probably run into some practical problems if you tried to nail it down, like the fact that for many people they are office or factory workers and the amount of time they spend outside doesn't change all that radically during summer, certainly not enough to make the difference between explosive spread and total eradication. "The more people are already infected, the fewer and further between the as-yet-uninfected are, so that seems quite reasonable" That's why the wave slows down yes, but again, this ad-hoc notion doesn't work. Try it out on paper or code up a quick epi model yourself and then compare against real world case data. The waves always end long before predicted, even when there are still tons of uninfected people available to infect and people wandering around who should easily infect them. This is one of the reasons why COVID modelling failed so badly. They coded up the exact simple germ theory model you're describing here, and of course it yields a single giant wave whereas what is seen in reality is a long series of small waves that start and end in ways that aren't predicted by the theory. "See first point above." See my reply. I can assure you, that the epidemiology of influenza really isn't as simple as "people going outside in summer". Take a look at some of the research papers from the 80s exploring this topic. There are still plenty of people interacting with each other closely indoors even in summer months, yet flu completely disappears. It's not something that ties in any obvious way back to accepted theory. "Someone always flies in supplies..." That's not what "totally isolated" means. In the cases in question there was no contact with the outside world whatsoever. In one case, at a British polar base, they mounted a very thorough investigation after an outbreak of cold virus after 17 weeks of total isolation. They checked if any new supply crates had been opened, etc, but no. They couldn't identify anywhere that new viruses might have been introduced to the base. https://www.cambridge.org/core/journals/epidemiology-and-infection/article/an-outbreak-of-common-colds-at-an-antarctic-base-after-seventeen-weeks-of-complete-isolation/1D3A49463583D06CEACE1CCF9C1A25B4 https://www.cambridge.org/core/journals/epidemiology-and-inf... One of the more popular sub-theories (though largely ignored by epidemiologists) to try and explain these things is the possibility that many people are continuously infected at a sub-symptomatic level, that the immune system never 100% wipes out the viral infection in these people, just keeps it in check. Then something happens to slightly knock the immune system out of balance for a moment, like a sudden change in temperature, and the infection is able to re-gain a foothold and starts replicating out of control again. So that'd be why people just show up everywhere at once spontaneously infected without any obvious index cases.