6 ms·

Here's how they handled the exposure vs. control group: Mice in the CO2-exposed group were housed within a semi-sealed, custom-made 393-L exposure chamber... t

Here's how they handled the exposure vs. control group: Mice in the CO2-exposed group were housed within a semi-sealed, custom-made 393-L exposure chamber... the exposure chamber was also fitted with two 80-mm fans that ran continuously... Mice in the control group were housed in the same type of cages in the same room, but not within the exposure chamber. So the exposure group were in a semi-sealed chamber where other toxins might have accumulated, and were subjected to continuous fan noise. Whereas the control group were not in a chamber and were not subjected to fan noise. I don't think one can draw any valid conclusions from this experiment. They should have put both groups in the same type of chamber, both with fans, but just not introduced CO2 into the chamber for the control group.

VectorLock 5y ago
Sounds like someone could simply recreate that experiment if they felt those variables had any bearing on the result.
- throwaway316943 5y ago
  Yes and they should do so before anyone draws any conclusions from this.
Etheryte 5y ago
I don't really see how this hand-wavy dismissal invalidates anything about the study. Both groups of mice were in the same room, and while they would be exposed to somewhat different levels of noise due to relative distance to the fans, the difference would be negligible. Same for the toxin buildup, the very reason the fans were present was to circulate the air, which was constantly both monitored and regulated. This was a core part of the whole setup and the main focus of the study. Unless you find it more likely that the small background noise difference had a starker effect on the lungs than the mixture of gases going through the lungs constantly, I don't really see how any of the criticisms hold.
- radford-neal 5y ago
  The fans were to circulate air within the chamber (because they worried, I think incorrectly, that the CO2 might sink to the bottom). There was nothing to stop the smell of urine, and other volatiles, such as outgassing from plastics, from accumulating within the chamber. There's no reason to think the difference in noise was negligible, and noise can have a big effect on things like stress levels, which can have wide-ranging health effects. Plus, there may well have been other effects of the chamber, such as different light levels, which could also have unknown health effects. It's absolutely fundamental for an experiment like this to try to match the exposure and control conditions as closely as possible, aside from the one variable you're looking at. They failed to do this, so their results are worthless.
  - 1e-9 5y ago
    > There was nothing to stop the smell of urine, and other volatiles, such as outgassing from plastics, from accumulating within the chamber. The paper states that the substrate and changing schedule had been shown to keep ammonia to low levels. > There's no reason to think the difference in noise was negligible... Actually, there is. A good 80 mm fan can push more than 10 cubic feet per minute at less than 10 dBA. That's far quieter than a human whisper or rustling leaves. It's comparable to the sound of a pin dropping. If mice are adversely affected by that, I don't see how they could survive the wild. > Plus, there may well have been other effects of the chamber, such as different light levels, which could also have unknown health effects. That seems like pure speculation.
    - radford-neal 5y ago
      What about things other than ammonia? And did they have a "good" fan, or a bad fan? The point of making the environment for the exposure and control groups be as near identical as possible is to equalize not only the factors you thought of, but also the factors you didn't think of. It's the idea that there were no important differences between exposure and control groups, despite their obviously different living environments, that is "pure speculation".
DoingIsLearning 5y ago
> So the exposure group were in a semi-sealed chamber where other toxins might have accumulated, and were subjected to continuous fan noise. Whereas the control group were not in a chamber and were not subjected to fan noise. It's fine to be critical of methodology, but if they are testing for lung function and tissue compliance, then I am struggling to find a reason for fan noise to be a lurking variable in this case.
- ben_w 5y ago
  Depends on the effect of general stress from the environment? Cortisol isn’t exactly a great thing for us to marinate in. (That is way out of my field of expertise, I don’t know how to properly gauge the relative strength of effect of stress hormones vs. CO2)
- krastanov 5y ago
  Frequently, in experiments done before "double blind" and "control group" became the standard, we had results where something for which we were "struggling to find a reason" to be important, actually turned out to be important. In a properly done experiment we would not need to have a conversation in which lay people guess what else might have affected the results.
beowulfey 5y ago
What does fan noise have to do with lung growth? What “toxins” are you predicting would accumulate? This comment seems insightful on the surface, but does not make much sense in the context of the experiment. I would say they used a valid control here. There are plenty of better reasons to be cautious on their conclusions. Personally, upon skimming the paper, I found it curious that male mice never saw significant effects in these experiments whereas females were weakly significant for only a few of these. Sometimes opposite trends were observed between males and females. That could be due to the difference in sample size (30 female vs 23 male). It also might only be affecting female mice, which would be unexpected (for me). But the differences and effects are not strong, and I am not at all convinced. I also don’t like that they separated males and females here because I would not expect there to be differences in lung development between the sexes (do we have different lungs? I am not a physiologist so I definitely could be wrong on this) and I try and avoid designs that require two-way ANOVA. Anyway, no need to bash the cage design, which is misleading I think, when the data themselves are as weak as they are.
- saiya-jin 5y ago
  while generally agreeing the experiment setup is far from perfect, I can imagine a number of physical manifestations of constant stress exposure. Heck, animals and people die from it if strong/persistent enough. If a person can turn their dark hair grey overnight by exposure to massive stress, other effects can be present too.
- bigfudge 5y ago
  Did you notice if they pre-registered the separate analyses of males and females? Flexibility in the analysis of data ("researcher degrees of freedom") is likely one of the major causes of the replication crisis. Gelman's paper is excellent in showing how these issues manifest without any intended malfeasance: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
- nick__m 5y ago
  I suggest anyone doubting the importance of eliminating all differences between the experimental cohort and the control group to read that speech from Feynman: https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm or this article that highlights it's most important part: https://www.realclearscience.com/blog/2014/02/the_rat_experiment_you_dont_know_about_but_should.html https://www.realclearscience.com/blog/2014/02/the_rat_experi... . If I had to reduce that to some phrases I would say : the easiest way to do bad science is to lie to yourself, so to do good science you must foremost learn how not to do that. And avoiding avoidable difference in cohorts is one of the first step to stay honest.
hanoz 5y ago
Why, time and time again, do we see results of major studies by career scientists which don't pass a moment's scrutiny by the educated layman?
- iratewizard 5y ago
  Have career scientists been held accountable for the replication crisis? I know Hanlon's razor says not to, but at this point I can't help but think medical and social science have been more or less taken over to produce propaganda. Not too dissimilar to what the CIA did during the Cold War.
  - bigfudge 5y ago
    It really doesn't need to be that complicated. From what I can see (in Psychology) a combination of statistical illiteracy and motivated cognition is more than enough.
- Robotbeat 5y ago
  Let’s just assume for the moment that the experiment is actually invalid (due to poor control group). Experiments that have good control groups don’t show spectacular results. Only those experiments that have spectacular results get publicized by the popular media. Therefore you are seeing a kind of bias particular more to the popular media than necessarily the scientific field.
leereeves 5y ago
Even more troubling, they completely screwed up the analysis, making a mistake that even undergraduates know to avoid. They measured 12 factors (as shown in table 1) across 4 treatment groups (male/female in two generations separately). That's 48 variables, so with the p<0.05 significance level they used, they would, on average, see 2.4 "statistically significant" results just by random chance. If I'm counting correctly, they found 2 "statistically significant" results, exactly as expected by random chance. In other words, their results are statistically indistinguishable from chance, exactly like the famous "jelly beans cause acne" thought experiment. https://www.explainxkcd.com/wiki/index.php/882:_Significant https://www.explainxkcd.com/wiki/index.php/882:_Significant Edit: Who's downvoting this without a response? Is there a mistake or is it simply inconvenient for the people who badly want to see an effect?
- 1e-9 5y ago
  Yes, it’s not valid to reject a null hypothesis involving multiple tests without using a multiple testing correction. The simplest conservative approach would be to use a Bonferroni correction and check whether any of the M tests has a p-value below alpha/M. Clearly, these results are not that significant. The study suggests a couple potential effects. To credibly evaluate them, the study needs to be repeated with more mice and/or fewer tests along with appropriate statistical analysis.

VectorLock 5y ago

Sounds like someone could simply recreate that experiment if they felt those variables had any bearing on the result.

throwaway316943 5y ago
Yes and they should do so before anyone draws any conclusions from this.

Etheryte 5y ago

I don't really see how this hand-wavy dismissal invalidates anything about the study. Both groups of mice were in the same room, and while they would be exposed to somewhat different levels of noise due to relative distance to the fans, the difference would be negligible. Same for the toxin buildup, the very reason the fans were present was to circulate the air, which was constantly both monitored and regulated. This was a core part of the whole setup and the main focus of the study. Unless you find it more likely that the small background noise difference had a starker effect on the lungs than the mixture of gases going through the lungs constantly, I don't really see how any of the criticisms hold.

radford-neal 5y ago
The fans were to circulate air within the chamber (because they worried, I think incorrectly, that the CO2 might sink to the bottom). There was nothing to stop the smell of urine, and other volatiles, such as outgassing from plastics, from accumulating within the chamber. There's no reason to think the difference in noise was negligible, and noise can have a big effect on things like stress levels, which can have wide-ranging health effects. Plus, there may well have been other effects of the chamber, such as different light levels, which could also have unknown health effects. It's absolutely fundamental for an experiment like this to try to match the exposure and control conditions as closely as possible, aside from the one variable you're looking at. They failed to do this, so their results are worthless.
- 1e-9 5y ago
  > There was nothing to stop the smell of urine, and other volatiles, such as outgassing from plastics, from accumulating within the chamber. The paper states that the substrate and changing schedule had been shown to keep ammonia to low levels. > There's no reason to think the difference in noise was negligible... Actually, there is. A good 80 mm fan can push more than 10 cubic feet per minute at less than 10 dBA. That's far quieter than a human whisper or rustling leaves. It's comparable to the sound of a pin dropping. If mice are adversely affected by that, I don't see how they could survive the wild. > Plus, there may well have been other effects of the chamber, such as different light levels, which could also have unknown health effects. That seems like pure speculation.
  - radford-neal 5y ago
    What about things other than ammonia? And did they have a "good" fan, or a bad fan? The point of making the environment for the exposure and control groups be as near identical as possible is to equalize not only the factors you thought of, but also the factors you didn't think of. It's the idea that there were no important differences between exposure and control groups, despite their obviously different living environments, that is "pure speculation".

DoingIsLearning 5y ago

> So the exposure group were in a semi-sealed chamber where other toxins might have accumulated, and were subjected to continuous fan noise. Whereas the control group were not in a chamber and were not subjected to fan noise. It's fine to be critical of methodology, but if they are testing for lung function and tissue compliance, then I am struggling to find a reason for fan noise to be a lurking variable in this case.

ben_w 5y ago
Depends on the effect of general stress from the environment? Cortisol isn’t exactly a great thing for us to marinate in. (That is way out of my field of expertise, I don’t know how to properly gauge the relative strength of effect of stress hormones vs. CO2)
krastanov 5y ago
Frequently, in experiments done before "double blind" and "control group" became the standard, we had results where something for which we were "struggling to find a reason" to be important, actually turned out to be important. In a properly done experiment we would not need to have a conversation in which lay people guess what else might have affected the results.

beowulfey 5y ago

What does fan noise have to do with lung growth? What “toxins” are you predicting would accumulate? This comment seems insightful on the surface, but does not make much sense in the context of the experiment. I would say they used a valid control here. There are plenty of better reasons to be cautious on their conclusions. Personally, upon skimming the paper, I found it curious that male mice never saw significant effects in these experiments whereas females were weakly significant for only a few of these. Sometimes opposite trends were observed between males and females. That could be due to the difference in sample size (30 female vs 23 male). It also might only be affecting female mice, which would be unexpected (for me). But the differences and effects are not strong, and I am not at all convinced. I also don’t like that they separated males and females here because I would not expect there to be differences in lung development between the sexes (do we have different lungs? I am not a physiologist so I definitely could be wrong on this) and I try and avoid designs that require two-way ANOVA. Anyway, no need to bash the cage design, which is misleading I think, when the data themselves are as weak as they are.

saiya-jin 5y ago
while generally agreeing the experiment setup is far from perfect, I can imagine a number of physical manifestations of constant stress exposure. Heck, animals and people die from it if strong/persistent enough. If a person can turn their dark hair grey overnight by exposure to massive stress, other effects can be present too.
bigfudge 5y ago
Did you notice if they pre-registered the separate analyses of males and females? Flexibility in the analysis of data ("researcher degrees of freedom") is likely one of the major causes of the replication crisis. Gelman's paper is excellent in showing how these issues manifest without any intended malfeasance: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
nick__m 5y ago
I suggest anyone doubting the importance of eliminating all differences between the experimental cohort and the control group to read that speech from Feynman: https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm or this article that highlights it's most important part: https://www.realclearscience.com/blog/2014/02/the_rat_experiment_you_dont_know_about_but_should.html https://www.realclearscience.com/blog/2014/02/the_rat_experi... . If I had to reduce that to some phrases I would say : the easiest way to do bad science is to lie to yourself, so to do good science you must foremost learn how not to do that. And avoiding avoidable difference in cohorts is one of the first step to stay honest.

hanoz 5y ago

Why, time and time again, do we see results of major studies by career scientists which don't pass a moment's scrutiny by the educated layman?

iratewizard 5y ago
Have career scientists been held accountable for the replication crisis? I know Hanlon's razor says not to, but at this point I can't help but think medical and social science have been more or less taken over to produce propaganda. Not too dissimilar to what the CIA did during the Cold War.
- bigfudge 5y ago
  It really doesn't need to be that complicated. From what I can see (in Psychology) a combination of statistical illiteracy and motivated cognition is more than enough.
Robotbeat 5y ago
Let’s just assume for the moment that the experiment is actually invalid (due to poor control group). Experiments that have good control groups don’t show spectacular results. Only those experiments that have spectacular results get publicized by the popular media. Therefore you are seeing a kind of bias particular more to the popular media than necessarily the scientific field.

leereeves 5y ago

Even more troubling, they completely screwed up the analysis, making a mistake that even undergraduates know to avoid. They measured 12 factors (as shown in table 1) across 4 treatment groups (male/female in two generations separately). That's 48 variables, so with the p<0.05 significance level they used, they would, on average, see 2.4 "statistically significant" results just by random chance. If I'm counting correctly, they found 2 "statistically significant" results, exactly as expected by random chance. In other words, their results are statistically indistinguishable from chance, exactly like the famous "jelly beans cause acne" thought experiment. https://www.explainxkcd.com/wiki/index.php/882:_Significant https://www.explainxkcd.com/wiki/index.php/882:_Significant Edit: Who's downvoting this without a response? Is there a mistake or is it simply inconvenient for the people who badly want to see an effect?

1e-9 5y ago
Yes, it’s not valid to reject a null hypothesis involving multiple tests without using a multiple testing correction. The simplest conservative approach would be to use a Bonferroni correction and check whether any of the M tests has a p-value below alpha/M. Clearly, these results are not that significant. The study suggests a couple potential effects. To credibly evaluate them, the study needs to be repeated with more mice and/or fewer tests along with appropriate statistical analysis.