6 ms·

Emily Oster has a useful post about why, despite the effort, its is difficult to control for everything that matters in observational studies: https://emilyoste

by mjb 4y ago

Emily Oster has a useful post about why, despite the effort, its is difficult to control for everything that matters in observational studies: https://emilyoster.substack.com/p/coffee-definitely-either-kills-you?s=w https://emilyoster.substack.com/p/coffee-definitely-either-k... Specifically: > As result, in the data, sugar starts to look…a lot worse for you over time. In the earliest period of the data there is no observed correlation between the sugar measure and BMI (see below) but by the latest period, it’s hugely correlated. The “sugar-makes-you-fat” story only shows up in the last few years, coincidentally during the period when sugar is less consumed by individuals who exercise, do not smoke and are better educated.

TameAntelope 4y ago
Yes, science is really hard, and it’s a good point by Oster, but you’ve got to think about the idea that the folks who conducted this study know as well how hard science is, and for them to publish this anyway means they (and their reviewers) did a pretty good job dealing with this issue (and many others). None of this is conclusive to the point where no further study is necessary, but I’m guessing this study will make some institutions think/rethink their guidance on coffee consumption.
- jaybaxter 4y ago
  >but you’ve got to think about the idea that the folks who conducted this study know as well how hard science is, and for them to publish this anyway means they (and their reviewers) did a pretty good job dealing with this issue (and many others). What makes you say that about this paper specifically? Certainly in general there are career incentives for authors to publish papers which are not very conclusive, and we know that relatively useless observational studies do get published.
- rajin444 4y ago
  > for them to publish this anyway means they did a pretty good job The standard shouldn’t be “we did the best we can” for the sake of publishing. It should be “is what we’re doing useful”. > I’m guessing this study will make some institutions think/rethink their guidance on coffee consumption. This is the major issue with all these kinds of studies. The absolute last thing they should be doing is influencing public policy. Data that’s more or less garbage but you treat it like it’s solid because it’s “science” is harmful.
  - TameAntelope 4y ago
    I'm sorry but I completely disagree; the standard absolutely should be "we're doing the best that is currently possible to determine how things work". Asking for more is entitled and completely misunderstands the nature of science. We need to have a way to figure out what to tell people to do, and we're doing that, whether you like it or not, with imperfect information. Also not for nothing, this data is not "more or less garbage". This is about as good as we can do in science right now, and honestly your tone is pretty terrible. It sounds like you want religious levels of certainty and you feel you deserve that. Science doesn't get much more certain than this (and this is far from very certain in any sense), there is no "better" way to do things that the researchers just lazily didn't follow here, and your suggesting that such is the case is ignorant and selfish. You should feel bad for what you've written here.
    - jtc331 4y ago
      > Science doesn't get much more certain than this A well designed RCT isn’t much more certain than this? Observational studies are hardly the peak for scientific knowledge.
      - guelo 4y ago
        A decades long rigorously controlled RCT in human nutrition science is practically impossible. Even if it were possible the scientist would still have to choose variables to control for in their population selection.
    - thereisnospork 4y ago
      You don't get points for trying. If observational 'scientists' can't discern something about the world which stands up to scrutiny then they are wasting everyone's time, including theirs.
PheonixPharts 4y ago
It's also important that people understand what "control for" means in an observational study. Nearly all observational studies that you read draw their conclusions from regression analysis (or something, such as a ANOVA, which is a can be trivially implemented as regression analysis). This just means that a linear model such as this is used: outcome ~ b_coffee * drinks_coffee + b_income_level * income_level .... Then, unlike in machine learning, researchers look at the value of the coefficients and most importantly the confidence in the coefficient value. In a simplified example: years_of_life ~ b_coffee * drinks_coffee + b_intercept Where b_intercept will represent the life expectancy in general. Then b_coffee (since drinks coffee is binary here) will represent whether or not coffee adds to your years of life. If it's negative it means drinking coffee reduces your life expectancy and if it's positive it means it increases it. In statistics we also look at how certain we are in b_coffee in terms of standard error and p-values. For example if b_coffee is say 5, meaning it adds 5 years to your life, but our standard error in this estimate is 4 (ie 95% chance that the real impact is between -3 and 13 years). In this case the p-value for this coefficient will be higher than necessary to conclude "statistical significance". But suppose the standard error is very small, like 1 year, where we are virtually certain that coffee does improve life expectancy. To control for say, college education, we just add that coefficient to our model. years_of_life ~ b_coffee * drinks_coffee + b_college + has_degree + b_intercept The magic of regression analysis is that if, in fact, people that go to college life longer and people that go to college also drink more coffee, the our coefficient for b_coffee will change in this new model to reflect this. If for example it were to become negative now (with a low p-value) what we would conclude from this model is that coffee is in fact bad for you, and college is good for you and it just happens that a lot of people who go to college also drink coffee. Just as even the most advanced AI is often just a lot of matrix multiplication with non-linear transforms, the vast majority of observational studies are drawing conclusion using linear models. In practice, as you add more variables to your model, you tend to get a lot of tricky to interpret results. Regression analysis is a remarkably powerful tool, but it is important to remember when you see publications this is what is really happening and there is a lot of room for subtlety when interpreting these models.
- vorpalhex 4y ago
  Yet all you have to do is miss a factor and you've ended up with correlation. As a simple question, what about people who have weak stomachs or hearts? My mother doesn't drink coffee because it "makes her heart beat too hard". How, with no actual medical data for "coffee makes my heart beat hard", do you control for that? Is that something to control for? This is the "caffeine and healthy pregnancy" problem. We know women who consume less than ~200mg of caffeine tend to have healthier pregnancies.. but if you can drink 5+ cups of coffee while pregnant and not get overtaken with nausea, that might indicate something is already wrong.
rhacker 4y ago
I'm kinda "done" with observational studies. It seems like for the last 10 years that's all we get. And every time the studies come out they contradict that last one. When are we going to see the actual coffee atoms attack bad viruses and shit (kinda a joke but seriously, when are we going to have that visibility into exact pathologies).
- rajin444 4y ago
  That kind of work is hard, expensive, and likely to be fruitless. And in the case of many softer sciences it would more or less remove the need for the field (in their current forms). Sociology is a band aid until we understand biology better. That being said I agree with you entirely and we shouldn’t shy away from hard problems because of this.
  - AussieWog93 4y ago
    >That kind of work is hard, expensive, and likely to be fruitless. Don't forget that the "fruit" for scientists is fame, not the truth. Anything that can get headlines and conference talks is a good thing, regardless of whether or not the methodology even makes sense.
    - jemorya 4y ago
      >Don't forget that the "fruit" for scientists is fame, not the truth. This is a pretty shocking thing to say. It's not true whatsoever for the field of physics. Is there a particular field or a particular experience you're reacting to? I think I'm overreacting to how general your statement is.
      - harry8 4y ago
        Feynmann's classic "Cargo Cult Science" for anyone who hasn't seen it yet. And for those of us who think it should be read and re-read at least annually and the time has come: https://sites.cs.ucsb.edu/~ravenben/cargocult.html https://sites.cs.ucsb.edu/~ravenben/cargocult.html It's a 5 minute read. Arguably the 5 most intellectually productive minutes we can spend. (Well at least for me, anyway). Now that ESP, paranormal, telekinesis research is thoroughly dead, (exposed by magicians like James Randi?), psychology still remains mired in it all. To the credit of a minority in that field they're having a go at getting it back to science and I wish them all the luck in the world. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis My understanding is that for vast amounts of "What food is healthy, what kills you." Research, epidemiological studies is all we have. They're obviously limited, easy to get wrong, easy to fool yourself (and you're the easiest person to fool!) The supply of identical twins, who are willing to commit to life long diet differences with rigor and make all the same choices outside of the study is, well, kinda low. The statistics being used for these things is still under active development and being improved. Can it ever be done properly? Well I guess so. I think we're pretty clear on the smoking, cancer, heart disease, stroke link nowadays, right? And those studies have to have been similar. It's interesting that scientists have to raise funding, publish or perish and so on to even have a career at all making them part P.T. Barnum. Fenymann, for all he didn't need to do that because of the different era, reflected "glory" from Los Alamos etc. really was capable of putting P.T. Barnum to shame while at the same time being the most devout adherent to and proselytiser of scientific principle & purity. So good and so lucky he could keep his hands clean? Is there no academic misconduct in physics nowadays? None? I'd believe you if you told me so & why.

by mjb 4y ago

TameAntelope 4y ago

Yes, science is really hard, and it’s a good point by Oster, but you’ve got to think about the idea that the folks who conducted this study know as well how hard science is, and for them to publish this anyway means they (and their reviewers) did a pretty good job dealing with this issue (and many others). None of this is conclusive to the point where no further study is necessary, but I’m guessing this study will make some institutions think/rethink their guidance on coffee consumption.

jaybaxter 4y ago
>but you’ve got to think about the idea that the folks who conducted this study know as well how hard science is, and for them to publish this anyway means they (and their reviewers) did a pretty good job dealing with this issue (and many others). What makes you say that about this paper specifically? Certainly in general there are career incentives for authors to publish papers which are not very conclusive, and we know that relatively useless observational studies do get published.
rajin444 4y ago
> for them to publish this anyway means they did a pretty good job The standard shouldn’t be “we did the best we can” for the sake of publishing. It should be “is what we’re doing useful”. > I’m guessing this study will make some institutions think/rethink their guidance on coffee consumption. This is the major issue with all these kinds of studies. The absolute last thing they should be doing is influencing public policy. Data that’s more or less garbage but you treat it like it’s solid because it’s “science” is harmful.
- TameAntelope 4y ago
  I'm sorry but I completely disagree; the standard absolutely should be "we're doing the best that is currently possible to determine how things work". Asking for more is entitled and completely misunderstands the nature of science. We need to have a way to figure out what to tell people to do, and we're doing that, whether you like it or not, with imperfect information. Also not for nothing, this data is not "more or less garbage". This is about as good as we can do in science right now, and honestly your tone is pretty terrible. It sounds like you want religious levels of certainty and you feel you deserve that. Science doesn't get much more certain than this (and this is far from very certain in any sense), there is no "better" way to do things that the researchers just lazily didn't follow here, and your suggesting that such is the case is ignorant and selfish. You should feel bad for what you've written here.
  - jtc331 4y ago
    > Science doesn't get much more certain than this A well designed RCT isn’t much more certain than this? Observational studies are hardly the peak for scientific knowledge.
    - guelo 4y ago
      A decades long rigorously controlled RCT in human nutrition science is practically impossible. Even if it were possible the scientist would still have to choose variables to control for in their population selection.
  - thereisnospork 4y ago
    You don't get points for trying. If observational 'scientists' can't discern something about the world which stands up to scrutiny then they are wasting everyone's time, including theirs.

PheonixPharts 4y ago

It's also important that people understand what "control for" means in an observational study. Nearly all observational studies that you read draw their conclusions from regression analysis (or something, such as a ANOVA, which is a can be trivially implemented as regression analysis). This just means that a linear model such as this is used: outcome ~ b_coffee * drinks_coffee + b_income_level * income_level .... Then, unlike in machine learning, researchers look at the value of the coefficients and most importantly the confidence in the coefficient value. In a simplified example: years_of_life ~ b_coffee * drinks_coffee + b_intercept Where b_intercept will represent the life expectancy in general. Then b_coffee (since drinks coffee is binary here) will represent whether or not coffee adds to your years of life. If it's negative it means drinking coffee reduces your life expectancy and if it's positive it means it increases it. In statistics we also look at how certain we are in b_coffee in terms of standard error and p-values. For example if b_coffee is say 5, meaning it adds 5 years to your life, but our standard error in this estimate is 4 (ie 95% chance that the real impact is between -3 and 13 years). In this case the p-value for this coefficient will be higher than necessary to conclude "statistical significance". But suppose the standard error is very small, like 1 year, where we are virtually certain that coffee does improve life expectancy. To control for say, college education, we just add that coefficient to our model. years_of_life ~ b_coffee * drinks_coffee + b_college + has_degree + b_intercept The magic of regression analysis is that if, in fact, people that go to college life longer and people that go to college also drink more coffee, the our coefficient for b_coffee will change in this new model to reflect this. If for example it were to become negative now (with a low p-value) what we would conclude from this model is that coffee is in fact bad for you, and college is good for you and it just happens that a lot of people who go to college also drink coffee. Just as even the most advanced AI is often just a lot of matrix multiplication with non-linear transforms, the vast majority of observational studies are drawing conclusion using linear models. In practice, as you add more variables to your model, you tend to get a lot of tricky to interpret results. Regression analysis is a remarkably powerful tool, but it is important to remember when you see publications this is what is really happening and there is a lot of room for subtlety when interpreting these models.

vorpalhex 4y ago
Yet all you have to do is miss a factor and you've ended up with correlation. As a simple question, what about people who have weak stomachs or hearts? My mother doesn't drink coffee because it "makes her heart beat too hard". How, with no actual medical data for "coffee makes my heart beat hard", do you control for that? Is that something to control for? This is the "caffeine and healthy pregnancy" problem. We know women who consume less than ~200mg of caffeine tend to have healthier pregnancies.. but if you can drink 5+ cups of coffee while pregnant and not get overtaken with nausea, that might indicate something is already wrong.

rhacker 4y ago

I'm kinda "done" with observational studies. It seems like for the last 10 years that's all we get. And every time the studies come out they contradict that last one. When are we going to see the actual coffee atoms attack bad viruses and shit (kinda a joke but seriously, when are we going to have that visibility into exact pathologies).

rajin444 4y ago
That kind of work is hard, expensive, and likely to be fruitless. And in the case of many softer sciences it would more or less remove the need for the field (in their current forms). Sociology is a band aid until we understand biology better. That being said I agree with you entirely and we shouldn’t shy away from hard problems because of this.
- AussieWog93 4y ago
  >That kind of work is hard, expensive, and likely to be fruitless. Don't forget that the "fruit" for scientists is fame, not the truth. Anything that can get headlines and conference talks is a good thing, regardless of whether or not the methodology even makes sense.
  - jemorya 4y ago
    >Don't forget that the "fruit" for scientists is fame, not the truth. This is a pretty shocking thing to say. It's not true whatsoever for the field of physics. Is there a particular field or a particular experience you're reacting to? I think I'm overreacting to how general your statement is.
    - harry8 4y ago
      Feynmann's classic "Cargo Cult Science" for anyone who hasn't seen it yet. And for those of us who think it should be read and re-read at least annually and the time has come: https://sites.cs.ucsb.edu/~ravenben/cargocult.html https://sites.cs.ucsb.edu/~ravenben/cargocult.html It's a 5 minute read. Arguably the 5 most intellectually productive minutes we can spend. (Well at least for me, anyway). Now that ESP, paranormal, telekinesis research is thoroughly dead, (exposed by magicians like James Randi?), psychology still remains mired in it all. To the credit of a minority in that field they're having a go at getting it back to science and I wish them all the luck in the world. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis My understanding is that for vast amounts of "What food is healthy, what kills you." Research, epidemiological studies is all we have. They're obviously limited, easy to get wrong, easy to fool yourself (and you're the easiest person to fool!) The supply of identical twins, who are willing to commit to life long diet differences with rigor and make all the same choices outside of the study is, well, kinda low. The statistics being used for these things is still under active development and being improved. Can it ever be done properly? Well I guess so. I think we're pretty clear on the smoking, cancer, heart disease, stroke link nowadays, right? And those studies have to have been similar. It's interesting that scientists have to raise funding, publish or perish and so on to even have a career at all making them part P.T. Barnum. Fenymann, for all he didn't need to do that because of the different era, reflected "glory" from Los Alamos etc. really was capable of putting P.T. Barnum to shame while at the same time being the most devout adherent to and proselytiser of scientific principle & purity. So good and so lucky he could keep his hands clean? Is there no academic misconduct in physics nowadays? None? I'd believe you if you told me so & why.