6 ms·

As I tell my students all the time -- properly-executed, an experiment that doesn't work is not a failure. If you don't internalize this idea, you will find ex

by ISL 7y ago

As I tell my students all the time -- properly-executed, an experiment that doesn't work is not a failure. If you don't internalize this idea, you will find experimental physics to be psychologically untenable. If you build an experiment (at any scale) to test out an idea that you think is going to work, and it doesn't work, you made progress. Now you know something really essential -- that thing that seemed like a well-motivated idea has some kind of unknown flaw. With that in hand, you can proceed to try something better, whether it is a refined approach or something entirely new. Without doing the experiment, that well-motivated idea would still look like something worth trying. If a large team of people were willing to spend years and lots of resources on an Alzheimer's clinical trial, it was likely to be because they thought it had a real chance of working. If it didn't work, we all benefit from their effort.

Fr0styMatt88 7y ago
This is a teaching that really stuck with me from one of my university professors as well. I really wish this was something that was taught more to the general public, I think especially in schools. At least for me, it's something I'd never heard said in a succinct way until I was in university.
- ImaCake 7y ago
  I wish this idea would sink in for scientists! Academia (my experience is in biological sciences) has a long tradition of hiding negative results and only valuing the rare positive results in a sea of messy data and unexplainable failure.
  - theseadroid 7y ago
    I doubt many scientists don't know this. However the current situation is largely if not all motivated by how funding is provided. Instead of a culture issue, I'd say it's a money issue.
duxup 7y ago
I always thought that was sort of the point of the business side of "Fail Fast". Try it now, if it doesn't work, now you know something valuable and you can try something else. As someone who suffers from a lot of analysis paralysis, distraction ... sometimes I find just making myself keep typing and trying things in order to fail quickly and learn from it leads to much faster / sooner successes rather than burying myself up in all the minuta and every aspect of something and... not doing it. There was an old TV show called Connections where they tracked various inventions and progress over history, and IIRC it is telling that almost every positive step they note involve someone failing to do something else they were trying, but still finding out something valuable that adds up to the next step for something else and so on. https://en.wikipedia.org/wiki/Connections_(TV_series) https://en.wikipedia.org/wiki/Connections_(TV_series)
- vikramkr 7y ago
  Unfortunately, even though these amyloid drugs keep on failing, the "try something else" part of the "if you fail, then you've learned something and can try something else" equation doesn't seem to really register with the industry for whatever reason. We keep pouring money into amyloid trials when it's pretty clear that there's still a lot of basic research to be done to find new leads to pursue.
  - duxup 7y ago
    It's also probably not encouraging that the Connections TV show spans ... often hundreds of years, or more. A lot of us are just falling for someone else.
hprotagonist 7y ago
The issue here is that for the past 25 years, multiple teams have spent years and lots of resources on the same idea despite it failing to work every previous time. Negative results are valuable. The same negative result over and over starts to feel a lot more like sunk-cost fallacy.
- vikramkr 7y ago
  On the other hand, if you test the same hypothesis again and again and again, and don't finish up a trial and use interim data readouts to inform decisions, then after enough time, by pure chance, you'll apparently be able to pull an effect size out of the air and convince wall street and apparently even maybe regulators that your drug now works! I really hope the FDA tells biogen to take a hike. The last thing people need is false hope, and the last thing we need as a society is to be spending billions of dollars on an alzheimers drug that doesn't work. For something that important (and that profitable), the least biogen could have done is run that trial through the end.
  - hprotagonist 7y ago
    From a previous post by the blogger on a separate drug washing out: And I think I’ll add this every time we have a big drug trial wipe out: keep in mind, that under a safety-only regulatory regime, that people would have been taking this drug (and paying for it, one way or another) for the last few years now, and it would have done them no good at all. When they could have been actually taking something that might help them. We do not know enough about investigational drugs to approve them without efficacy data.
    - vikramkr 7y ago
      And the great thing about safety-and-efficacy regimes is that until you get a drug to work, the market can't be filled. If the FDA makes exceptions for biogen, then the alzheimers market for new drugs shrinks, disincentivizing research into drugs that will actually work. Why invest billions to make drugs that work if you can make that same money by pumping trash into the market?
dekhn 7y ago
I've worked in both experimental physics and experimental biology (specifically in drug discovery, an early phase that can lead to clinical trials). Failed experiments in biology don't make much progress compared to failed experiments in physics. beyond the inability to control for wide range of variables that are normally managed in physics, most people in biology aren't good at running high quality, reproducible experiments that falsify hypothesis. I have seen many coworkers run the same exact gel over and over until they get one that "looks like what they were expecting". I don't see the same level of experimental inadequacy in physics. The best approaches in biology now seem to throw away the idea that we can construct a rational narrative to explain the biology that causes a complex disease, and more on collecting very large amounts of high quality data to produce sophisticated statistical models that deal with the underlying complexity in a useful way.
- vikramkr 7y ago
  ON the other hand, those methods of collecting large amounts of high quality data might only seem promising because they're effectively just ways to run thousands or millions of gels at once before picking out the ones that "look right." The fundamental incentives to not publish null results and to publish or perish haven't disappeared. It's not that people aren't capable of running high quality, reproducible experiments, or that the tools aren't there, it's just that people need to publish to advance in their careers. These new high throughput screens get your quick publications and let you overfit models to the data you collect, and it might seem like a transformation now, but if people weren't incentivized to spend a couple hundred bucks to replicate a gel in the past, do you think they'll actually be incentivized to spend thousands of dollars and computation time to try and replicate screens and the models derived from them?
  - killjoywashere 7y ago
    I gave a talk on basic bioinformatics algorithms to my pathologist colleagues once, explaining that it all amounts to collections of good guesses and you really need to know how this was done before you believe it. They just sat there, waited for the lecture to end, and went back to their jobs, accepting whatever the molecular labs were telling them that week. They simply have no mechanism to push back except to play gatekeeper, and they can't even be an informed gatekeeper.
agumonkey 7y ago
Reminds me how some rejections from HR felt like a motivator. When I could finally talk a bit about what was needed I could isolate the flaws and redesign the "theory". I wonder if it's passion that makes researcher pain-free regarding failure. Or if they also dreaded it but learned how to dodge or tame it.
derefr 7y ago
Ah, but if you "build an experiment (at any scale) to test out an idea that you think is going to work", you've already gone wrong. This is the Tao of the null hypothesis: if you build an experiment to test out an idea you think won't work, things can only go one of two ways—either you'll be proven right, or you'll discover something about the universe. Either way, you get to brag!
bigfudge 7y ago
Out of interest, roughly what proportion of experiments in physics "don't work"? Does data that fails to support the experimental hypothesis get published prominently in physics journals?
- knzhou 7y ago
  99% of experiments in fundamental physics "don't work" in this sense. All of them get published. If you got funded to build an apparatus, people are going to want to know what you saw. You can check for yourself, by looking at the experimental papers on the arXiv [0]. Every paper here that mentions data, but doesn't have the word "discovery" in the title, is a negative result. There are thousands of such papers. 0: https://arxiv.org/list/hep-ex/current https://arxiv.org/list/hep-ex/current
  - mortenjorck 7y ago
    I suppose you could think of it as an unexplored continent that has very specific requirements to survey each region. You mount an expedition, and you probably just find more trees and brush. The important thing, though, is that now that area is on a map, and the map keeps getting bigger.
- mirimir 7y ago
  I'm having no luck finding any URLs, but if you look at histories of many physical constants, you see that new measurements tend to lie within error ranges of the most recent previous measurements. But occasionally, there's a jump to a substantially different value. The explanation is that scientists debug their setups and equipment until they get a result that's plausible, relative to prior published measurements. Because otherwise, people would be driven to figure out what they did wrong. And it could be embarrassing. But once they do, they tend to stop tweaking, and publish. So it's only scientists who don't manage to get plausible results, and yet are confident that they've debugged well enough, who publish results that end up becoming new standards.
WalterBright 7y ago
Often failed experiments, in their failure, give an indication of an entirely new direction. Many groundbreaking discoveries are made that way. For example, the discovery of synthetic dyes came about from trying to synthesize quinine. https://en.wikipedia.org/wiki/William_Henry_Perkin https://en.wikipedia.org/wiki/William_Henry_Perkin The discovery of the vacuum tube was an accident resulting from Edison playing round with light bulbs trying to figure out why the filaments were breaking. https://en.wikipedia.org/wiki/Thermionic_emission https://en.wikipedia.org/wiki/Thermionic_emission

by ISL 7y ago

Fr0styMatt88 7y ago

This is a teaching that really stuck with me from one of my university professors as well. I really wish this was something that was taught more to the general public, I think especially in schools. At least for me, it's something I'd never heard said in a succinct way until I was in university.

ImaCake 7y ago
I wish this idea would sink in for scientists! Academia (my experience is in biological sciences) has a long tradition of hiding negative results and only valuing the rare positive results in a sea of messy data and unexplainable failure.
- theseadroid 7y ago
  I doubt many scientists don't know this. However the current situation is largely if not all motivated by how funding is provided. Instead of a culture issue, I'd say it's a money issue.

duxup 7y ago

I always thought that was sort of the point of the business side of "Fail Fast". Try it now, if it doesn't work, now you know something valuable and you can try something else. As someone who suffers from a lot of analysis paralysis, distraction ... sometimes I find just making myself keep typing and trying things in order to fail quickly and learn from it leads to much faster / sooner successes rather than burying myself up in all the minuta and every aspect of something and... not doing it. There was an old TV show called Connections where they tracked various inventions and progress over history, and IIRC it is telling that almost every positive step they note involve someone failing to do something else they were trying, but still finding out something valuable that adds up to the next step for something else and so on. https://en.wikipedia.org/wiki/Connections_(TV_series) https://en.wikipedia.org/wiki/Connections_(TV_series)

vikramkr 7y ago
Unfortunately, even though these amyloid drugs keep on failing, the "try something else" part of the "if you fail, then you've learned something and can try something else" equation doesn't seem to really register with the industry for whatever reason. We keep pouring money into amyloid trials when it's pretty clear that there's still a lot of basic research to be done to find new leads to pursue.
- duxup 7y ago
  It's also probably not encouraging that the Connections TV show spans ... often hundreds of years, or more. A lot of us are just falling for someone else.

hprotagonist 7y ago

The issue here is that for the past 25 years, multiple teams have spent years and lots of resources on the same idea despite it failing to work every previous time. Negative results are valuable. The same negative result over and over starts to feel a lot more like sunk-cost fallacy.

vikramkr 7y ago
On the other hand, if you test the same hypothesis again and again and again, and don't finish up a trial and use interim data readouts to inform decisions, then after enough time, by pure chance, you'll apparently be able to pull an effect size out of the air and convince wall street and apparently even maybe regulators that your drug now works! I really hope the FDA tells biogen to take a hike. The last thing people need is false hope, and the last thing we need as a society is to be spending billions of dollars on an alzheimers drug that doesn't work. For something that important (and that profitable), the least biogen could have done is run that trial through the end.
- hprotagonist 7y ago
  From a previous post by the blogger on a separate drug washing out: And I think I’ll add this every time we have a big drug trial wipe out: keep in mind, that under a safety-only regulatory regime, that people would have been taking this drug (and paying for it, one way or another) for the last few years now, and it would have done them no good at all. When they could have been actually taking something that might help them. We do not know enough about investigational drugs to approve them without efficacy data.
  - vikramkr 7y ago
    And the great thing about safety-and-efficacy regimes is that until you get a drug to work, the market can't be filled. If the FDA makes exceptions for biogen, then the alzheimers market for new drugs shrinks, disincentivizing research into drugs that will actually work. Why invest billions to make drugs that work if you can make that same money by pumping trash into the market?

dekhn 7y ago

I've worked in both experimental physics and experimental biology (specifically in drug discovery, an early phase that can lead to clinical trials). Failed experiments in biology don't make much progress compared to failed experiments in physics. beyond the inability to control for wide range of variables that are normally managed in physics, most people in biology aren't good at running high quality, reproducible experiments that falsify hypothesis. I have seen many coworkers run the same exact gel over and over until they get one that "looks like what they were expecting". I don't see the same level of experimental inadequacy in physics. The best approaches in biology now seem to throw away the idea that we can construct a rational narrative to explain the biology that causes a complex disease, and more on collecting very large amounts of high quality data to produce sophisticated statistical models that deal with the underlying complexity in a useful way.

vikramkr 7y ago
ON the other hand, those methods of collecting large amounts of high quality data might only seem promising because they're effectively just ways to run thousands or millions of gels at once before picking out the ones that "look right." The fundamental incentives to not publish null results and to publish or perish haven't disappeared. It's not that people aren't capable of running high quality, reproducible experiments, or that the tools aren't there, it's just that people need to publish to advance in their careers. These new high throughput screens get your quick publications and let you overfit models to the data you collect, and it might seem like a transformation now, but if people weren't incentivized to spend a couple hundred bucks to replicate a gel in the past, do you think they'll actually be incentivized to spend thousands of dollars and computation time to try and replicate screens and the models derived from them?
- killjoywashere 7y ago
  I gave a talk on basic bioinformatics algorithms to my pathologist colleagues once, explaining that it all amounts to collections of good guesses and you really need to know how this was done before you believe it. They just sat there, waited for the lecture to end, and went back to their jobs, accepting whatever the molecular labs were telling them that week. They simply have no mechanism to push back except to play gatekeeper, and they can't even be an informed gatekeeper.

agumonkey 7y ago

Reminds me how some rejections from HR felt like a motivator. When I could finally talk a bit about what was needed I could isolate the flaws and redesign the "theory". I wonder if it's passion that makes researcher pain-free regarding failure. Or if they also dreaded it but learned how to dodge or tame it.

derefr 7y ago

Ah, but if you "build an experiment (at any scale) to test out an idea that you think is going to work", you've already gone wrong. This is the Tao of the null hypothesis: if you build an experiment to test out an idea you think won't work, things can only go one of two ways—either you'll be proven right, or you'll discover something about the universe. Either way, you get to brag!

bigfudge 7y ago

Out of interest, roughly what proportion of experiments in physics "don't work"? Does data that fails to support the experimental hypothesis get published prominently in physics journals?

knzhou 7y ago
99% of experiments in fundamental physics "don't work" in this sense. All of them get published. If you got funded to build an apparatus, people are going to want to know what you saw. You can check for yourself, by looking at the experimental papers on the arXiv [0]. Every paper here that mentions data, but doesn't have the word "discovery" in the title, is a negative result. There are thousands of such papers. 0: https://arxiv.org/list/hep-ex/current https://arxiv.org/list/hep-ex/current
- mortenjorck 7y ago
  I suppose you could think of it as an unexplored continent that has very specific requirements to survey each region. You mount an expedition, and you probably just find more trees and brush. The important thing, though, is that now that area is on a map, and the map keeps getting bigger.
mirimir 7y ago
I'm having no luck finding any URLs, but if you look at histories of many physical constants, you see that new measurements tend to lie within error ranges of the most recent previous measurements. But occasionally, there's a jump to a substantially different value. The explanation is that scientists debug their setups and equipment until they get a result that's plausible, relative to prior published measurements. Because otherwise, people would be driven to figure out what they did wrong. And it could be embarrassing. But once they do, they tend to stop tweaking, and publish. So it's only scientists who don't manage to get plausible results, and yet are confident that they've debugged well enough, who publish results that end up becoming new standards.

WalterBright 7y ago

Often failed experiments, in their failure, give an indication of an entirely new direction. Many groundbreaking discoveries are made that way. For example, the discovery of synthetic dyes came about from trying to synthesize quinine. https://en.wikipedia.org/wiki/William_Henry_Perkin https://en.wikipedia.org/wiki/William_Henry_Perkin The discovery of the vacuum tube was an accident resulting from Edison playing round with light bulbs trying to figure out why the filaments were breaking. https://en.wikipedia.org/wiki/Thermionic_emission https://en.wikipedia.org/wiki/Thermionic_emission