6 ms·

These results from observational studies almost never replicate in randomized studies. It's because people who decide to take glucosamine (or not) also do lots

by Gatsky 4y ago

These results from observational studies almost never replicate in randomized studies. It's because people who decide to take glucosamine (or not) also do lots of other confounding things, like visit the doctor more, or have higher salaries, or don't live with people who smoke etc. It's been shown many times you can't really adjust for these things. Epidemiologists love doing these studies, because they are comparatively easy. The hardest part is the data sharing agreement with whatever registry you are using. You don't even need to leave your desk to collect any data.

mrfusion 4y ago
Good advice in general but it’s worth reading to see what confounders they tried to correct for. There’s still information to be gleaned. This part caught my eye: > A stronger association between glucosamine use and decreased lung cancer risk was observed in participants with a family history of lung cancer when compared with those without a family history.
- SoftTalker 4y ago
  A family history of lung cancer could simply mean a family history of smoking, which was much more common a few generations ago than it is today.
  - zoom6628 4y ago
    This was done in Guangzhou where I lived for years until 2018. The incidence of smoking amongst males still higher - gut guess of 30-40% adult males are smokers in the city and province. So in their cohort one would expect to be seeing still living relatives of young adult smokers who are also smokers. Interesting that it’s sold as supplement for joint and related tissue repair and yet first big science study with observable results is about cancer. Other interesting point is that it’s study of cancer. And that is simply because China still has huge numbers of smokers and the health cost is stratospheric.
    - a9h74j 4y ago
      To me it is plausible that in a time of great social and economic change, the odds of behavioral confounds in observational studies might increase. Non-stationarity everywhere. That said, a meaningful result here would be welcome.
    - SOLAR_FIELDS 4y ago
      Is it actually stratospheric though, when you take out QOL year adjustments? Other studies, such as this one in Finland[1], found that when you don’t take into account QOL adjustments for the smoker that smoking is actually a net benefit for society because the smokers die earlier and don’t draw on the very expensive late life healthcare. 1: https://bmjopen.bmj.com/content/2/6/e001678 https://bmjopen.bmj.com/content/2/6/e001678
adtac 4y ago
The real question is: why is anything other than randomised studies allowed to be published in the first place?
- yCombLinks 4y ago
  Observational studies are far cheaper and easier, and are actually very useful for a lot of situations.
- derbOac 4y ago
  Not questioning the need for RCTs, but they have their own issues. Like, let's say you do a RCT and it shows an effect, but then in a large epi observational study there isn't any? There would be some problems to work out. RCTs are great but can have problems with generalizability to real-world conditions. Ideally you'd study both.
  - q7xvh97o2pDhNrh 4y ago
    With a sufficiently robust RCT, I'd think that indicates an issue with the observation in the observational study -- i.e., there are some confounding factors in the real world -- rather than any sort of issue with whatever effect the RCT found. Conversely, if we had a large observational study that was then contradicted by a robust RCT... well, I'd still be inclined to trust the RCT. (At the same time, I think a good observational study is, by its nature, more applicable to one's own life. If an observational study suggests that $FOOBAR is good, then -- as long as there's also a solid RCT confirming that $FOOBAR is safe -- why not try out some $FOOBAR for yourself?)
- Gatsky 4y ago
  I guess it's ok to publish uninteresting and probably false things. This could be a fairly accurate description of biomedical science to be honest. It's annoying when it ends up on HN for example, or in the health section of a tabloid newspaper. In theory, these observational findings could suggest treatments that should be evaluated in a randomised trial. I am not actually sure this is true however, it is possible they are less useful than selecting interventions for further testing in some other way. Anecdotally, folk remedies may have a higher hit rate for selecting treatments for proper evaluation. This is often surprising - eg a precursor to aspirin is found in the Willow tree, which has been used as a medicine for thousands of years. Another example is the foxglove plant from which the heart drug digoxin is derived, which was used as a herbal medicine predating modern medicine.
dukeofdoom 4y ago
I agree with that. You can see this in Ontario COVID statistics. People that did not take the vaccine, despite all the pressure, tend to be healthier as population. They kind of self selected by being self confident in their health and not worrying about dying from COVID. The triple vaccinated in Ontario are now the highest risk of getting COVID. They too self selected, by tending to be older with pre existing health problems, and worried enough to get boosters. https://covid-19.ontario.ca/data?fbclid=1 https://covid-19.ontario.ca/data?fbclid=1
- ImPostingOnHN 4y ago
  does that include the people who died after not taking the vaccine? asking because you only linked to the homepage of a general data portal which itself reports its no longer being updated, versus to a study supporting what you're saying
  - dukeofdoom 4y ago
    Last update was June 10. So it's still pretty recent. Ontario did not release data on death rates to the public. I don't know why. I briefly read on twitter that UK is showing startling differences now, with the vaccinated having a much higher all cause mortality rate. If true, I don't know if its this effect at play or if some of the conspiracy theories were right.
    - ImPostingOnHN 4y ago
      just reiterating in case you missed it, regardless of when it was last updated: you actually only linked to the homepage of a general data portal (not even a specific data set), versus to a study supporting what you're saying I have to imagine this is a mistake in getting the specific URL you were actually looking at
- ramraj07 4y ago
  You are right that people who take vaccines and who do not probably bifurcate along other lines as well. But you’re doing everything even more of a disservice by speculating the dimensions as if it’s established fact. Maybe the unvaccinated aren’t healthier they’re just liars? Also any self respecting study would control for the simple demographics like age so your hypothesis would be quite moot if the study was done with any rigor.
vcdimension 4y ago
Actually, you're wrong about that. According to this Cochrane systematic review, RCT's and observational studies are usually in close agreement: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR000034.pub2/full https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR...
- rich_sasha 4y ago
  Hmm, that would be interesting, except it seems they only considered 14 studies. They find in 11 of these observational studies match the results of RCTs. Im sure this will be different in areas where there is no public awareness of a risk factor vs ones where there are. Eg healthy diets select for people who care about their health, so it's not so infomative, vs something where the link is not publicly known (so that the observational study is already somewhat blind/randomized).
- Gatsky 4y ago
  Well no, I’m not wrong. That article you linked is a review of reviews comparing RCTs and observational studies for medical interventions, primarily surgical studies or pharmacological treatments with well characterised effects. Observational studies of supplements and nutritional interventions on the other hand have a very poor track record of RCT validation.
  - vcdimension 4y ago
    @Gatsky: ah I see, yes that sounds highly plausible, but can you supply any evidence of this, or is it just hearsay?
- gojomo 4y ago
  Such a review is necessarily limited to topics which have both an RCT and an observational study. As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement. (It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.) The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.
stoorafa 4y ago
You’re right that there is non-random allocation to the exposure (taking glucosamine), but this is something that can be addressed to an extent with a bit of statistics. I was disappointed to see the authors didn’t take the additional step of doing propensity score matching or weighting to account for propensity to take glucosamine. Here’s a good reference about propensity score matching: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144483/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144483/
JoelMcCracken 4y ago
I wonder, does anyone know of a good guide for lay persons that explains how to understand scientific studies? Even something that details the difference between a population study and randomized controlled trials.
- Jedd 4y ago
  Ben Goldacre's book 'Bad Pharma' has a good discussion around this, especially in terms of caveats and gotchas to watch out for with such studies. It might be more 'lay person' than you are looking for, but it's an excellent read for myriad other reasons, so you wouldn't be wasting your time (IMO).
  - JoelMcCracken 4y ago
    Thanks. I was hoping for something online, like a blog post/website, which is an easy way to spread the word to other people so they know how to contextualize these things.

by Gatsky 4y ago

mrfusion 4y ago

Good advice in general but it’s worth reading to see what confounders they tried to correct for. There’s still information to be gleaned. This part caught my eye: > A stronger association between glucosamine use and decreased lung cancer risk was observed in participants with a family history of lung cancer when compared with those without a family history.

SoftTalker 4y ago
A family history of lung cancer could simply mean a family history of smoking, which was much more common a few generations ago than it is today.
- zoom6628 4y ago
  This was done in Guangzhou where I lived for years until 2018. The incidence of smoking amongst males still higher - gut guess of 30-40% adult males are smokers in the city and province. So in their cohort one would expect to be seeing still living relatives of young adult smokers who are also smokers. Interesting that it’s sold as supplement for joint and related tissue repair and yet first big science study with observable results is about cancer. Other interesting point is that it’s study of cancer. And that is simply because China still has huge numbers of smokers and the health cost is stratospheric.
  - a9h74j 4y ago
    To me it is plausible that in a time of great social and economic change, the odds of behavioral confounds in observational studies might increase. Non-stationarity everywhere. That said, a meaningful result here would be welcome.
  - SOLAR_FIELDS 4y ago
    Is it actually stratospheric though, when you take out QOL year adjustments? Other studies, such as this one in Finland[1], found that when you don’t take into account QOL adjustments for the smoker that smoking is actually a net benefit for society because the smokers die earlier and don’t draw on the very expensive late life healthcare. 1: https://bmjopen.bmj.com/content/2/6/e001678 https://bmjopen.bmj.com/content/2/6/e001678

adtac 4y ago

The real question is: why is anything other than randomised studies allowed to be published in the first place?

yCombLinks 4y ago
Observational studies are far cheaper and easier, and are actually very useful for a lot of situations.
derbOac 4y ago
Not questioning the need for RCTs, but they have their own issues. Like, let's say you do a RCT and it shows an effect, but then in a large epi observational study there isn't any? There would be some problems to work out. RCTs are great but can have problems with generalizability to real-world conditions. Ideally you'd study both.
- q7xvh97o2pDhNrh 4y ago
  With a sufficiently robust RCT, I'd think that indicates an issue with the observation in the observational study -- i.e., there are some confounding factors in the real world -- rather than any sort of issue with whatever effect the RCT found. Conversely, if we had a large observational study that was then contradicted by a robust RCT... well, I'd still be inclined to trust the RCT. (At the same time, I think a good observational study is, by its nature, more applicable to one's own life. If an observational study suggests that $FOOBAR is good, then -- as long as there's also a solid RCT confirming that $FOOBAR is safe -- why not try out some $FOOBAR for yourself?)
Gatsky 4y ago
I guess it's ok to publish uninteresting and probably false things. This could be a fairly accurate description of biomedical science to be honest. It's annoying when it ends up on HN for example, or in the health section of a tabloid newspaper. In theory, these observational findings could suggest treatments that should be evaluated in a randomised trial. I am not actually sure this is true however, it is possible they are less useful than selecting interventions for further testing in some other way. Anecdotally, folk remedies may have a higher hit rate for selecting treatments for proper evaluation. This is often surprising - eg a precursor to aspirin is found in the Willow tree, which has been used as a medicine for thousands of years. Another example is the foxglove plant from which the heart drug digoxin is derived, which was used as a herbal medicine predating modern medicine.

dukeofdoom 4y ago

I agree with that. You can see this in Ontario COVID statistics. People that did not take the vaccine, despite all the pressure, tend to be healthier as population. They kind of self selected by being self confident in their health and not worrying about dying from COVID. The triple vaccinated in Ontario are now the highest risk of getting COVID. They too self selected, by tending to be older with pre existing health problems, and worried enough to get boosters. https://covid-19.ontario.ca/data?fbclid=1 https://covid-19.ontario.ca/data?fbclid=1

ImPostingOnHN 4y ago
does that include the people who died after not taking the vaccine? asking because you only linked to the homepage of a general data portal which itself reports its no longer being updated, versus to a study supporting what you're saying
- dukeofdoom 4y ago
  Last update was June 10. So it's still pretty recent. Ontario did not release data on death rates to the public. I don't know why. I briefly read on twitter that UK is showing startling differences now, with the vaccinated having a much higher all cause mortality rate. If true, I don't know if its this effect at play or if some of the conspiracy theories were right.
  - ImPostingOnHN 4y ago
    just reiterating in case you missed it, regardless of when it was last updated: you actually only linked to the homepage of a general data portal (not even a specific data set), versus to a study supporting what you're saying I have to imagine this is a mistake in getting the specific URL you were actually looking at
ramraj07 4y ago
You are right that people who take vaccines and who do not probably bifurcate along other lines as well. But you’re doing everything even more of a disservice by speculating the dimensions as if it’s established fact. Maybe the unvaccinated aren’t healthier they’re just liars? Also any self respecting study would control for the simple demographics like age so your hypothesis would be quite moot if the study was done with any rigor.

vcdimension 4y ago

Actually, you're wrong about that. According to this Cochrane systematic review, RCT's and observational studies are usually in close agreement: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR000034.pub2/full https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR...

rich_sasha 4y ago
Hmm, that would be interesting, except it seems they only considered 14 studies. They find in 11 of these observational studies match the results of RCTs. Im sure this will be different in areas where there is no public awareness of a risk factor vs ones where there are. Eg healthy diets select for people who care about their health, so it's not so infomative, vs something where the link is not publicly known (so that the observational study is already somewhat blind/randomized).
Gatsky 4y ago
Well no, I’m not wrong. That article you linked is a review of reviews comparing RCTs and observational studies for medical interventions, primarily surgical studies or pharmacological treatments with well characterised effects. Observational studies of supplements and nutritional interventions on the other hand have a very poor track record of RCT validation.
- vcdimension 4y ago
  @Gatsky: ah I see, yes that sounds highly plausible, but can you supply any evidence of this, or is it just hearsay?
gojomo 4y ago
Such a review is necessarily limited to topics which have both an RCT and an observational study. As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement. (It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.) The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.

stoorafa 4y ago

You’re right that there is non-random allocation to the exposure (taking glucosamine), but this is something that can be addressed to an extent with a bit of statistics. I was disappointed to see the authors didn’t take the additional step of doing propensity score matching or weighting to account for propensity to take glucosamine. Here’s a good reference about propensity score matching: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144483/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144483/

JoelMcCracken 4y ago

I wonder, does anyone know of a good guide for lay persons that explains how to understand scientific studies? Even something that details the difference between a population study and randomized controlled trials.

Jedd 4y ago
Ben Goldacre's book 'Bad Pharma' has a good discussion around this, especially in terms of caveats and gotchas to watch out for with such studies. It might be more 'lay person' than you are looking for, but it's an excellent read for myriad other reasons, so you wouldn't be wasting your time (IMO).
- JoelMcCracken 4y ago
  Thanks. I was hoping for something online, like a blog post/website, which is an easy way to spread the word to other people so they know how to contextualize these things.