8 ms·

When I started reading the article I thought the title was a bit click-baity for Science. But then I noticed that the theory it refers to was not the amyloid hy

When I started reading the article I thought the title was a bit click-baity for Science. But then I noticed that the theory it refers to was not the amyloid hypothesis itself, but the toxic oligomer hypothesis that emerged later. That theory is pretty much the main potential explanation on why every single drug targeting Aβ failed that still keeps Aβ fibrils relevant. It's a very convenient theory because it keeps the main original observations about the fibrils relevant while explaining why therapies that target them don't work. One part that is really important that is mentioned in the middle of the article is that these systems are very difficult to handle, and it's almost impossible to make many of them nicely reproducible. Fibril and oligomer formation depends a lot on the environment and reacts to tiny differences. I find this kind of fraud deeply frustrating, there is so much wasted effort in the wake of faked high-profile results.

chmod600 4y ago
"very difficult to handle, and it's almost impossible to make many of them nicely reproducible" Those same qualities that make a study hard to reproduce should also create skepticism that the first study was done properly.
- fabian2k 4y ago
  It's more insidious than that. In a field where the core subject is difficult to handle it is nothing unusual if a different lab cannot recreate known experiments on their first try. This doesn't mean anything is wrong with the original research, it might just mean that the second lab didn't control all the variables. The part that is difficult and annoying is that the variables are not necessarily known, it takes a lot of time and experiments to actually nail down those. And even then for some sensitive stuff every detail can matter, like the exact type of tube you did the experiment in or the vendor, batch and age of every chemical you used. Very often you can control this enough to have consistent results for a set of experiments, but that kind of stuff is really hard to control across different labs.
  - LudwigNagasena 4y ago
    But is there a reason to privilege the first study results over others? Sounds like a case of the baby duck syndrome.
    - mxwsn 4y ago
      The social default is in some fields where work is hard to reproduce is they still get into good journals and are believed up front. Usually this is fine. A lot of work that is hard to reproduce, but are relatively important or open up many research directions, do eventually get reproduced, though it might take a year or more. And usually the first party does their best to try to help reproduce the results.
      - vlovich123 4y ago
        I wonder what it would look like if you’re not allowed to cite papers in the field unless there’s been 5 independent labs that have managed to reproduce the result and all the labs that reproduce get to share in the credit (eg if there’s a Nobel prize). Would that change the incentive structure and create better outcomes? Science isn’t science until you have a repeatable process in place where observations match outcomes and the underlying theory. Someone proposing a bold new theory without the ability to test it usually gets significant credit even if posthumously for being a bold visionary which seems to have been sufficient throughout time as a motivator for the really foundational scientific victories.
        toxik 4y ago
        The effort required to reproduce a result, especially an exotic one, is often prohibitive. The reward for successful reproduction is near zero, and if you fail, it’s often absolutely zero - you are not going to be able to contest a result by a failure. The original authors can simply say that your reproduction was flawed, if they need to answer at all. A journal that requires five confirmations to cite a work is a journal that will fast have zero submissions. The main reason people attempt reproduction is to continue that line of research, and that often starts with reproduction plus a slight tweak. This is why you find a lot of papers like this. This is also how irreproducible results become sort of known about but not challenged in many fields.
- buscoquadnary 4y ago
  It turns out applying those might not even be enough. https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
- szundi 4y ago
  If they wrote this in the article then bad interpretations are coming from readers. It makes sense to publish this, maybe someone else can make it stable and reproduce.
- spoonjim 4y ago
  I worked for a pharmaceutical company that tried to open a second factory producing more of the same thing that it was already successfully making and it never worked. They built the entire factory, it didn't work, they spent years and millions of dollars trying to debug it, it simply never worked out. They ended up just essentially running the US factory at three shifts to meet the increased demand. To this day I don't think anyone knows why the European production line never worked. Things can fail to reproduce without fraud.
  - vaylian 4y ago
    Very fascinating. Were you at least able to pinpoint a part of the production process that produced different results, or was the difference only visible once the final product was "assembled"?
    - spoonjim 4y ago
      I was not closely involved with the project. They essentially built the first one to "spec" like "use this kind of steel in the tanks" and the remediation process was essentially tearing those out one by one and "use the steel that we used in the first tank."
      - Aqwis 4y ago
        Sounds like a process similar to how you would debug a difficult bug in a big and/or complicated codebase – through trying to isolate the code causing the bug, comparing the buggy code with code that's known to work, replacing other parts of the code with dummy code/mocks etc. And sometimes a bug seems impossible to solve regardless of how much developer time you throw at it, and you just have to ship the product with the bug (seems to be especially common in the game dev business).
panabee 4y ago
as a non-scientist outsider, a few questions if you don't mind: 1. is it common for a non-reproduced experiment to gain prominence as a leading hypothesis and spur related research? 2. if so, why? if not, why did this hypothesis take hold? 3. how can we make science more reproducible? reproducibility is held as a tenet of science, but this example, assuming it is true, would violate the principle.
- londons_explore 4y ago
  Trust, but don't verify, because verification is expensive and unrewarding both financially and career-wise
- ethbr0 4y ago
  1. Yes. Science has a snowball effect as a result of committee-driven grant decisions. Research in a hot current topic attracts more grant funding more reliably. 2. It does because some fields of science are intrinsically long-duration, highly sensitive to variables, or only testable at scales that exceed our experimental capability. That's just a consequence of physics and natural laws of reality. E.g. nutrition, chemical toxicity, economics, high energy physics. 3. We can cheat and find proxies or more tenable micro-systems to experiment on. But often those have their own problems (e.g. rodent models) or aren't feasible. Reproducibility is a goal of science. It's not always an achievable goal. When it's not, we do the best we can, as with drug testing pipelines.
  - panabee 4y ago
    thanks for this response. given the prevalence of non-reproducibility, what other fields do you believe have suspect leading hypotheses?
    - Quekid5 4y ago
      Not the person you replied to, but... Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks. Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc. Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science. [0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).
galangalalgol 4y ago
What are the other hypothesis that would have been explored in the absence of the toxic oligamer papers?
AB674 4y ago
It's not even the toxic oligomer hypothesis that is undermined - it is only the role of amyloid beta as a primary driver of Alzheimer's pathology. This isn't even out of nowhere, there's been plenty of data finding no strong association between AB levels and pathology. Most data seems to point to AB being more of a supporting element, with tau amyloids actually being associated with toxicity and resultant functional deficits.

chmod600 4y ago

"very difficult to handle, and it's almost impossible to make many of them nicely reproducible" Those same qualities that make a study hard to reproduce should also create skepticism that the first study was done properly.

fabian2k 4y ago
It's more insidious than that. In a field where the core subject is difficult to handle it is nothing unusual if a different lab cannot recreate known experiments on their first try. This doesn't mean anything is wrong with the original research, it might just mean that the second lab didn't control all the variables. The part that is difficult and annoying is that the variables are not necessarily known, it takes a lot of time and experiments to actually nail down those. And even then for some sensitive stuff every detail can matter, like the exact type of tube you did the experiment in or the vendor, batch and age of every chemical you used. Very often you can control this enough to have consistent results for a set of experiments, but that kind of stuff is really hard to control across different labs.
- LudwigNagasena 4y ago
  But is there a reason to privilege the first study results over others? Sounds like a case of the baby duck syndrome.
  - mxwsn 4y ago
    The social default is in some fields where work is hard to reproduce is they still get into good journals and are believed up front. Usually this is fine. A lot of work that is hard to reproduce, but are relatively important or open up many research directions, do eventually get reproduced, though it might take a year or more. And usually the first party does their best to try to help reproduce the results.
    - vlovich123 4y ago
      I wonder what it would look like if you’re not allowed to cite papers in the field unless there’s been 5 independent labs that have managed to reproduce the result and all the labs that reproduce get to share in the credit (eg if there’s a Nobel prize). Would that change the incentive structure and create better outcomes? Science isn’t science until you have a repeatable process in place where observations match outcomes and the underlying theory. Someone proposing a bold new theory without the ability to test it usually gets significant credit even if posthumously for being a bold visionary which seems to have been sufficient throughout time as a motivator for the really foundational scientific victories.
      - toxik 4y ago
        The effort required to reproduce a result, especially an exotic one, is often prohibitive. The reward for successful reproduction is near zero, and if you fail, it’s often absolutely zero - you are not going to be able to contest a result by a failure. The original authors can simply say that your reproduction was flawed, if they need to answer at all. A journal that requires five confirmations to cite a work is a journal that will fast have zero submissions. The main reason people attempt reproduction is to continue that line of research, and that often starts with reproduction plus a slight tweak. This is why you find a lot of papers like this. This is also how irreproducible results become sort of known about but not challenged in many fields.
buscoquadnary 4y ago
It turns out applying those might not even be enough. https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
szundi 4y ago
If they wrote this in the article then bad interpretations are coming from readers. It makes sense to publish this, maybe someone else can make it stable and reproduce.
spoonjim 4y ago
I worked for a pharmaceutical company that tried to open a second factory producing more of the same thing that it was already successfully making and it never worked. They built the entire factory, it didn't work, they spent years and millions of dollars trying to debug it, it simply never worked out. They ended up just essentially running the US factory at three shifts to meet the increased demand. To this day I don't think anyone knows why the European production line never worked. Things can fail to reproduce without fraud.
- vaylian 4y ago
  Very fascinating. Were you at least able to pinpoint a part of the production process that produced different results, or was the difference only visible once the final product was "assembled"?
  - spoonjim 4y ago
    I was not closely involved with the project. They essentially built the first one to "spec" like "use this kind of steel in the tanks" and the remediation process was essentially tearing those out one by one and "use the steel that we used in the first tank."
    - Aqwis 4y ago
      Sounds like a process similar to how you would debug a difficult bug in a big and/or complicated codebase – through trying to isolate the code causing the bug, comparing the buggy code with code that's known to work, replacing other parts of the code with dummy code/mocks etc. And sometimes a bug seems impossible to solve regardless of how much developer time you throw at it, and you just have to ship the product with the bug (seems to be especially common in the game dev business).

panabee 4y ago

as a non-scientist outsider, a few questions if you don't mind: 1. is it common for a non-reproduced experiment to gain prominence as a leading hypothesis and spur related research? 2. if so, why? if not, why did this hypothesis take hold? 3. how can we make science more reproducible? reproducibility is held as a tenet of science, but this example, assuming it is true, would violate the principle.

londons_explore 4y ago
Trust, but don't verify, because verification is expensive and unrewarding both financially and career-wise
ethbr0 4y ago
1. Yes. Science has a snowball effect as a result of committee-driven grant decisions. Research in a hot current topic attracts more grant funding more reliably. 2. It does because some fields of science are intrinsically long-duration, highly sensitive to variables, or only testable at scales that exceed our experimental capability. That's just a consequence of physics and natural laws of reality. E.g. nutrition, chemical toxicity, economics, high energy physics. 3. We can cheat and find proxies or more tenable micro-systems to experiment on. But often those have their own problems (e.g. rodent models) or aren't feasible. Reproducibility is a goal of science. It's not always an achievable goal. When it's not, we do the best we can, as with drug testing pipelines.
- panabee 4y ago
  thanks for this response. given the prevalence of non-reproducibility, what other fields do you believe have suspect leading hypotheses?
  - Quekid5 4y ago
    Not the person you replied to, but... Everything is extremely hype-driven -- it turns out that "cannot confirm X" isn't very compelling for journals, etc. etc. or even the news cycle. Journals, etc. thrive on exciting new findings... and that tends to lessen the critical looks. Of course, there are lots of other things to this problem: Very narrow fields where people absolutely know who their "anonymous" paper reviewers will be, and so must include even extremely tangential references to those reviewers' papers, etc. etc. Usually the sciences self-correct eventually[0], but that's only because there is such a thing as objectively verifiable facts and overwhelming statistics in science. [0] Unfortunately often as slowly as "one funeral at a time" (Max Planck, I think).

galangalalgol 4y ago

What are the other hypothesis that would have been explored in the absence of the toxic oligamer papers?

AB674 4y ago

It's not even the toxic oligomer hypothesis that is undermined - it is only the role of amyloid beta as a primary driver of Alzheimer's pathology. This isn't even out of nowhere, there's been plenty of data finding no strong association between AB levels and pathology. Most data seems to point to AB being more of a supporting element, with tau amyloids actually being associated with toxicity and resultant functional deficits.