6 ms·

This reminds me of Feynman's story of the researcher who eliminated possible variables in rat experiments one by one, to find a surprising result (EDIT: To find

by cm2012 9y ago

This reminds me of Feynman's story of the researcher who eliminated possible variables in rat experiments one by one, to find a surprising result (EDIT: To find that rats didn't need smell, spatial relationships, taste, or vision to find the cheese - the sound of the floor was enough) https://www.realclearscience.com/blog/2014/02/the_rat_experiment_you_dont_know_about_but_should.html https://www.realclearscience.com/blog/2014/02/the_rat_experi...

nsnick 9y ago
Your comment is click bait.
- cm2012 9y ago
  Edited
vorotato 9y ago
TLDR: Long story with a moral of document your steps, and read others documentation lest you fall into the same traps they did.
gwern 9y ago
I never did manage to trace Feynman's story back to a source, but nevertheless, also something Derek Lowe repeatedly covers for biology/chemistry on "In the Pipeline": a lot of the 'cell lines' or 'reagents' used in lab work turn out to just not be what they're supposed to be, either because they never were or they degraded into something else in storage or something. So to give a recent paper he blogged, there's something like 30k papers using "immune system cells" which are actually just HeLas; presumably the cell lines in question got contaminated or swapped somewhere along the way, but that raises the question of what any of those papers mean... And of course there are similar issues with mice. You have all the different strains which are genetically different, but they'll also be genetically changing over time through drift & selection & mutations, so your Wistar-XXX may not be the Wistar-XXX from 50 years ago. Or mice chow - sure, the label may say it's the same but when was the last time you checked? (Think about that paper about how all sorts of lab animals, supposedly fed constant diets and under constant controlled environments, keep getting fatter over the past century. Something is changing; what? Genetics? Food composition? Micronutrients? Lab infections? Microbiomes? Measurement tools themselves?)
- jaggederest 9y ago
  I think it would be very interesting to have labs run detailed A-A studies to investigate these kind of things. "This kind of mouse we raised under these conditions, and we changed nothing, but they varied in these ways over time" I often wonder whether the variables we check against are actually as independent and subject to the central limit theorem as we assume.
  - gwern 9y ago
    You can run A/A studies in the sense of permutation tests or bootstraps where you shuffle individual mice which weren't treated and see how often you get a 'significant' result; this is how you can see stuff like litter or cage or room effects (ie mice housed close to each other will respond more similarly to each other, and woe betide you if you do your 'randomization' by selecting from different cages rather than selecting from different individuals!) eg "Design, power, and interpretation of studies in the standard murine model of ALS" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.654.7344&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.654... , Scott et al 2008, subsamples data from their 'control' mice to empirically estimate pseudo-effects and the null distribution. This is equivalent to raising two batches of mice and comparing. More interesting are multi-lab attempts at eliminating that heterogeneity and getting consistency in results, the most recent example being _C. elegans_ life extension - it turns out to be incredibly hard to harmonize labs in terms of measurements, environment, and results, and that's for well-established effects in model organisms which are about as simple as possible. Unsurprisingly, there's a ton of heterogeneity which means it's easy to violate your statistical assumptions (i.i.d. being more important than normality here, I think) and so you get a lot of false positives. It's similar to the brain imaging issue: the assumption of spatial locality and Gaussianness is not really true and drives a lot of false-positive _p_-value inflation. There are really quite a lot of problems with animal studies, so it's not a surprise that they wipe out spectacularly when transferring to human. I keep a list of systematic reviews & meta-analyses I run across at https://www.gwern.net/DNB-FAQ#fn95 https://www.gwern.net/DNB-FAQ#fn95 One of the most impressive ones, I think, are sepsis treatments: ~150 sepsis treatment experiments in humans based on mouse/rat experiments, and all of them failed: "Genomic responses in mouse models poorly mimic human inflammatory diseases", Seok et al 2013 http://www.pnas.org/content/110/9/3507.full.html http://www.pnas.org/content/110/9/3507.full.html
evincarofautumn 9y ago
I suppose humans tend to underuse, or undertrain, their senses, so we fail to consider the possibility that other animals might be taking advantage of their senses in different ways. In elementary school, I used to listen to the sounds of people’s pencils scratching on the desk, and could accurately determine what they were writing, especially if my head was near the desk and I knew what their handwriting looked like. Curves and lines, and strokes of different lengths and directions against the grain of the paper, have different frequency profiles; lifting and setting the pencil back down generates a tapping sound…that sort of thing. People often don’t believe me when I tell them this, because they figure that human hearing just isn’t that sensitive, or that the sound of handwriting doesn’t convey that much information—but it does! You’ve simply never noticed or paid attention to it because you had no desire or reason to. Another animal might, for various reasons. Once, my dad picked up my Magna-Doodle board as we stood outside my room, and started writing a message to me. I listened closely and said “…Clean your room?”—he had written “Cʟᴇᴀɴ Yᴏᴜʀ Rᴏᴏᴍ”—and I’ll never forget his perplexed look, first at me, then turning the board around to see if any light could be shining through. He still didn’t quite believe that I could do what I had just done in front of him!
- deftturtle 9y ago
  The force is strong with you
- flycaliguy 9y ago
  That's amazing. What about chalkboard writing? I imagine that would be louder, have more protracted strokes and the patterns could become more easily absorbed through watching the board. I should add though, that "clean your room" would be a pretty safe bet in most parent/child relationships.
  - evincarofautumn 9y ago
    My point is more that it’s not amazing, just something most people wouldn’t notice or think to try. Chalkboards were possible but harder, I think because of the distance—you don’t get the frequencies that don’t carry as well across a room. I could never really do it with whiteboards because they’re much quieter. And yeah, “clean your room” was definitely a safe bet, haha, then again we rely on context all the time when listening to speech too.
andai 9y ago
"They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running the rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about the rats." But what Young discovered, according to Feynman was infinitely more vital. "In fact, he discovered all the things you have to do to discover something about rats."

by cm2012 9y ago

nsnick 9y ago

Your comment is click bait.

cm2012 9y ago
Edited

vorotato 9y ago

TLDR: Long story with a moral of document your steps, and read others documentation lest you fall into the same traps they did.

gwern 9y ago

I never did manage to trace Feynman's story back to a source, but nevertheless, also something Derek Lowe repeatedly covers for biology/chemistry on "In the Pipeline": a lot of the 'cell lines' or 'reagents' used in lab work turn out to just not be what they're supposed to be, either because they never were or they degraded into something else in storage or something. So to give a recent paper he blogged, there's something like 30k papers using "immune system cells" which are actually just HeLas; presumably the cell lines in question got contaminated or swapped somewhere along the way, but that raises the question of what any of those papers mean... And of course there are similar issues with mice. You have all the different strains which are genetically different, but they'll also be genetically changing over time through drift & selection & mutations, so your Wistar-XXX may not be the Wistar-XXX from 50 years ago. Or mice chow - sure, the label may say it's the same but when was the last time you checked? (Think about that paper about how all sorts of lab animals, supposedly fed constant diets and under constant controlled environments, keep getting fatter over the past century. Something is changing; what? Genetics? Food composition? Micronutrients? Lab infections? Microbiomes? Measurement tools themselves?)

jaggederest 9y ago
I think it would be very interesting to have labs run detailed A-A studies to investigate these kind of things. "This kind of mouse we raised under these conditions, and we changed nothing, but they varied in these ways over time" I often wonder whether the variables we check against are actually as independent and subject to the central limit theorem as we assume.
- gwern 9y ago
  You can run A/A studies in the sense of permutation tests or bootstraps where you shuffle individual mice which weren't treated and see how often you get a 'significant' result; this is how you can see stuff like litter or cage or room effects (ie mice housed close to each other will respond more similarly to each other, and woe betide you if you do your 'randomization' by selecting from different cages rather than selecting from different individuals!) eg "Design, power, and interpretation of studies in the standard murine model of ALS" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.654.7344&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.654... , Scott et al 2008, subsamples data from their 'control' mice to empirically estimate pseudo-effects and the null distribution. This is equivalent to raising two batches of mice and comparing. More interesting are multi-lab attempts at eliminating that heterogeneity and getting consistency in results, the most recent example being _C. elegans_ life extension - it turns out to be incredibly hard to harmonize labs in terms of measurements, environment, and results, and that's for well-established effects in model organisms which are about as simple as possible. Unsurprisingly, there's a ton of heterogeneity which means it's easy to violate your statistical assumptions (i.i.d. being more important than normality here, I think) and so you get a lot of false positives. It's similar to the brain imaging issue: the assumption of spatial locality and Gaussianness is not really true and drives a lot of false-positive _p_-value inflation. There are really quite a lot of problems with animal studies, so it's not a surprise that they wipe out spectacularly when transferring to human. I keep a list of systematic reviews & meta-analyses I run across at https://www.gwern.net/DNB-FAQ#fn95 https://www.gwern.net/DNB-FAQ#fn95 One of the most impressive ones, I think, are sepsis treatments: ~150 sepsis treatment experiments in humans based on mouse/rat experiments, and all of them failed: "Genomic responses in mouse models poorly mimic human inflammatory diseases", Seok et al 2013 http://www.pnas.org/content/110/9/3507.full.html http://www.pnas.org/content/110/9/3507.full.html

evincarofautumn 9y ago

I suppose humans tend to underuse, or undertrain, their senses, so we fail to consider the possibility that other animals might be taking advantage of their senses in different ways. In elementary school, I used to listen to the sounds of people’s pencils scratching on the desk, and could accurately determine what they were writing, especially if my head was near the desk and I knew what their handwriting looked like. Curves and lines, and strokes of different lengths and directions against the grain of the paper, have different frequency profiles; lifting and setting the pencil back down generates a tapping sound…that sort of thing. People often don’t believe me when I tell them this, because they figure that human hearing just isn’t that sensitive, or that the sound of handwriting doesn’t convey that much information—but it does! You’ve simply never noticed or paid attention to it because you had no desire or reason to. Another animal might, for various reasons. Once, my dad picked up my Magna-Doodle board as we stood outside my room, and started writing a message to me. I listened closely and said “…Clean your room?”—he had written “Cʟᴇᴀɴ Yᴏᴜʀ Rᴏᴏᴍ”—and I’ll never forget his perplexed look, first at me, then turning the board around to see if any light could be shining through. He still didn’t quite believe that I could do what I had just done in front of him!

deftturtle 9y ago
The force is strong with you
flycaliguy 9y ago
That's amazing. What about chalkboard writing? I imagine that would be louder, have more protracted strokes and the patterns could become more easily absorbed through watching the board. I should add though, that "clean your room" would be a pretty safe bet in most parent/child relationships.
- evincarofautumn 9y ago
  My point is more that it’s not amazing, just something most people wouldn’t notice or think to try. Chalkboards were possible but harder, I think because of the distance—you don’t get the frequencies that don’t carry as well across a room. I could never really do it with whiteboards because they’re much quieter. And yeah, “clean your room” was definitely a safe bet, haha, then again we rely on context all the time when listening to speech too.

andai 9y ago

"They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running the rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about the rats." But what Young discovered, according to Feynman was infinitely more vital. "In fact, he discovered all the things you have to do to discover something about rats."