4 ms·

In physics, we have been aggressively publishing negative results for decades. There is an entire field dedicated to such work, called "Physics Beyond the Stand

In physics, we have been aggressively publishing negative results for decades. There is an entire field dedicated to such work, called "Physics Beyond the Standard Model" (spoiler: there isn't any). I've seen entire careers of extremely good experimentalists dedicated to "failing to reject the null hypothesis" at more an more stringent limits (neutrinoless double beta decay is a good example of this) and have been to week-long conferences where every single paper was either a crazy theory or a negative experimental result. I left the field 15 years ago because I didn't want to spend my career measuring zero, and presumably over time it will eventually dry up. The conditions for its existence seem to be more to do with having a highly trained group of people who have exhausted all plausible avenues of research in a given area and are left chasing a few scraps. In areas where there are still plenty of positive results to be had, the tendency will always be to emphasize the positive. As a partial solution to this tendency, in my applied physics work, where I did get positive results, I tried to include a section in papers entitled "Things That Didn't Work So Well" that sketched failed approaches to save other people the trouble of trying stuff that seemed like a good idea but didn't pan out... at the very least we should expect that from the average publication, and be suspicious of any experimental paper that does not include some description of the blind alleys.

ISL 12y ago
A counterpoint -- neutrino oscillation, dark matter, dark energy, and gravity all exist outside the standard model. If we are to believe that there's some larger theory that ties everything together, then there's something missing. That we have done such extensive searches without finding anything is an amazing property of nature. The growing body of experimental null searches serve to provide tighter and tighter constraints on our understanding of nature. If you put your best new idea to the test, and it fails, it's not a failure, it's a success. Now you know your best idea, the one you thought could lead you forward, is a dead end. Your next steps will be better-guided and take you further forward.
- eru 12y ago
  Yes. It's just that biologists don't think like this (yet?).
bravura 12y ago
Keep in mind that there's a difference between a negative result and failure to find. I don't know enough about your field to understand the results you're talking about, but in machine learning "failing to reject the null hypothesis" generally indicates failure to find, not a genuine negative result. I find strong negative results to be compelling, but have little interest in wading through failure to find. Aside: In NLP + ML, there was a now defunct publication called the Journal For Interesting Negative Results: http://jinr.org/ http://jinr.org/
specialist 12y ago
What you describe sounds (to this layperson) like a solid QA/Test strategy. I strongly approve.
surement 12y ago
> at the very least we should expect that from the average publication, and be suspicious of any experimental paper that does not include some description of the blind alleys. In mathematics, being suspicious might be a bit much, but it's a very valuable (for your audience) habit to have. For example, the book Introduction to the Theory of Computation by Michael Sipser does this a lot: it often explores avenues that lead to dead ends before arriving at the correct proof. This is the way mathematicians work, and it's usually hidden from papers or other textbooks. I'm guessing that this is either for economy or because authors want to look like the first thing they tried was the right one.
- alexbecker 12y ago
  Another excellent example of this is the paper "How Not to Prove the Poincare Conjecture".
Balgair 12y ago
"have been to week-long conferences where every single paper was either a crazy theory or a negative experimental result" That sounds incredibly fun. The crazier and zanier the better. Were there awards for 'Most likely to actually be random words', those are my favorites. Also, I just started in neuroscience from an EE/Physics background. You would be AMAZED at what gets published. The rigor of physics is just.... incomprehensible in bio. Granted, yes, the experiments are 'squishy' and hard to quantify. You run a gel, and you get signal or not, counting it hard enough as is, let alone trying to measure anything. Still, the papers we read.... It's like they tried to do math, but gave up halfway through. I've seen papers where they start out with 2 decimal accuracy and errorbars in Figure 1, drop the errors completely in figure 2, and then just go to counting by 10s in Figure 4. Somehow, they derive significant results that journals will publish with only 20 cells or something. Mind you, nowhere do they quote the temperature, the pressure, what their saline solution actually is, how many failure experiments they ran, etc. The physicist in me is just ... pain. And oh god, the egos. Papers will have just one author on them, the PI, and when you look at the webpage, 20 faces show up for the postdocs alone in that lab. I have no idea where they get these fools, but they seem to be in large supply. Feynman had a lecture (please help, I can't find the link) about what it takes to get a rat to not smell food behind a door. Turns out, it's a lot of work. I've been in smell labs that just plain ignore the research, even when they know it's there. "It's too much work and funding, besides, the data should suss out the real mechanisms." The McGill study earlier this year on pain in male rats being modulated by the sex of the experimenter? Yep, just plain ignored as well. I'll share some links here on the issues in bio, it's a lot. https://en.m.wikipedia.org/wiki/John_P._A._Ioannidis https://en.m.wikipedia.org/wiki/John_P._A._Ioannidis http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble http://www.economist.com/news/briefing/21588057-scientists-t...
- deleted 12y ago
  [deleted]
- deleted 12y ago
  [deleted]
- shalmanese 12y ago
  "All experiments in psychology are not of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on--with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before. The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell. He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell. Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers that clues that the rat is really using-- not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running." From: http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html
jerf 12y ago
Particle physics is the light shining in the darkness of science. I understand that not all sciences can be particle physics, but most scientists could stand to learn a lesson or two about it.

ISL 12y ago

A counterpoint -- neutrino oscillation, dark matter, dark energy, and gravity all exist outside the standard model. If we are to believe that there's some larger theory that ties everything together, then there's something missing. That we have done such extensive searches without finding anything is an amazing property of nature. The growing body of experimental null searches serve to provide tighter and tighter constraints on our understanding of nature. If you put your best new idea to the test, and it fails, it's not a failure, it's a success. Now you know your best idea, the one you thought could lead you forward, is a dead end. Your next steps will be better-guided and take you further forward.

eru 12y ago
Yes. It's just that biologists don't think like this (yet?).

bravura 12y ago

Keep in mind that there's a difference between a negative result and failure to find. I don't know enough about your field to understand the results you're talking about, but in machine learning "failing to reject the null hypothesis" generally indicates failure to find, not a genuine negative result. I find strong negative results to be compelling, but have little interest in wading through failure to find. Aside: In NLP + ML, there was a now defunct publication called the Journal For Interesting Negative Results: http://jinr.org/ http://jinr.org/

specialist 12y ago

What you describe sounds (to this layperson) like a solid QA/Test strategy. I strongly approve.

surement 12y ago

> at the very least we should expect that from the average publication, and be suspicious of any experimental paper that does not include some description of the blind alleys. In mathematics, being suspicious might be a bit much, but it's a very valuable (for your audience) habit to have. For example, the book Introduction to the Theory of Computation by Michael Sipser does this a lot: it often explores avenues that lead to dead ends before arriving at the correct proof. This is the way mathematicians work, and it's usually hidden from papers or other textbooks. I'm guessing that this is either for economy or because authors want to look like the first thing they tried was the right one.

alexbecker 12y ago
Another excellent example of this is the paper "How Not to Prove the Poincare Conjecture".

Balgair 12y ago

"have been to week-long conferences where every single paper was either a crazy theory or a negative experimental result" That sounds incredibly fun. The crazier and zanier the better. Were there awards for 'Most likely to actually be random words', those are my favorites. Also, I just started in neuroscience from an EE/Physics background. You would be AMAZED at what gets published. The rigor of physics is just.... incomprehensible in bio. Granted, yes, the experiments are 'squishy' and hard to quantify. You run a gel, and you get signal or not, counting it hard enough as is, let alone trying to measure anything. Still, the papers we read.... It's like they tried to do math, but gave up halfway through. I've seen papers where they start out with 2 decimal accuracy and errorbars in Figure 1, drop the errors completely in figure 2, and then just go to counting by 10s in Figure 4. Somehow, they derive significant results that journals will publish with only 20 cells or something. Mind you, nowhere do they quote the temperature, the pressure, what their saline solution actually is, how many failure experiments they ran, etc. The physicist in me is just ... pain. And oh god, the egos. Papers will have just one author on them, the PI, and when you look at the webpage, 20 faces show up for the postdocs alone in that lab. I have no idea where they get these fools, but they seem to be in large supply. Feynman had a lecture (please help, I can't find the link) about what it takes to get a rat to not smell food behind a door. Turns out, it's a lot of work. I've been in smell labs that just plain ignore the research, even when they know it's there. "It's too much work and funding, besides, the data should suss out the real mechanisms." The McGill study earlier this year on pain in male rats being modulated by the sex of the experimenter? Yep, just plain ignored as well. I'll share some links here on the issues in bio, it's a lot. https://en.m.wikipedia.org/wiki/John_P._A._Ioannidis https://en.m.wikipedia.org/wiki/John_P._A._Ioannidis http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble http://www.economist.com/news/briefing/21588057-scientists-t...

deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
shalmanese 12y ago
"All experiments in psychology are not of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on--with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before. The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell. He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell. Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers that clues that the rat is really using-- not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running." From: http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html

jerf 12y ago

Particle physics is the light shining in the darkness of science. I understand that not all sciences can be particle physics, but most scientists could stand to learn a lesson or two about it.