5 ms·

This is a cohort study conducted over 15 years. You can't really criticize it only based on those numbers, at least not without looking at similar numbers from

This is a cohort study conducted over 15 years. You can't really criticize it only based on those numbers, at least not without looking at similar numbers from other cohort studies with the same length. I tend to ascribe some value to "It was published in the peer-reviewed British Medical Journal." Additional thought: Especially because it is essentially supporting the null hypothesis.

flatline 8y ago
There is nothing more frustrating than doing things you are bad at, regularly, especially if you are bound to get somewhat worse as time goes on. To your point, even if that were the reason for a significant number of departures, it would not go against the conclusion.
- ido 8y ago
  Sounds like a use case ripe for game developers to jump on to.
joe_the_user 8y ago
You can't really criticize it only based on those numbers, at least not without looking at similar numbers from other cohort studies with the same length. Well, one can't criticize the quality of the study without looking how well other studies did. But even if all studies involve a massive drop-out rate, the drop-out rate still seems like something to look at in evaluating the accuracy of the study. I mean my concern, looking from the outside, isn't relative accuracy, but absolute accuracy. Not "is it as good as other studies?" but "it is good enough that I can rely on it?"
- mjevans 8y ago
  15 -years- is a LONG time for a study. I'm not sure about the dropout rates, but it is a real statistical issue. I'm just not sure there's anything that can reasonably be done about it. You'd have to have something like federally mandated participation / continued participation. That's also very unlikely to pass ethics boards. BTW, it would also be an issue if the subjects were compensated to the point that they differed from the actual population at large. Fundamentally having more respect for doing things right, the right way, over the /long/ term, including the STEM fields, would be a 'good thing' for any society, but particularly western ones (where often the bad/evil antagonist is based on science or a cold lack of emotions; but the protagonist is based mostly on emotion and feeling instead of logic).
  - joe_the_user 8y ago
    15 -years- is a LONG time for a study. I'm not sure about the dropout rates, but it is a real statistical issue. I'm just not sure there's anything that can reasonably be done about it. Everyone keeps replying as if my comments imply a demand that "something be done about this". That's not my point. Maybe nothing can be done about it. The situation is still significant or at least appears significant.
    - b_tterc_p 8y ago
      Similarly, it may be that running studies like this is really difficult and that massively reduced sample sizes are something we must accept for practical reasons... but that doesn’t mean it permits us to ignore statistical ramifications
- digitalzombie 8y ago
  > "it is good enough that I can rely on it?" It's called longitudinal analysis or survival analysis, depending on the question being asked. We statistician got test and models for these. You can rely on it if the people who are conducting the experiment are competent and know what they're doing.
nolemurs 8y ago
> This is a cohort study conducted over 15 years. You can't really criticize it only based on those numbers You absolutely can. Scientific studies should be judged on their potential to produce interesting results. This study has too small a sample size to provide results that merit making even the weakest claims about. Even the initial sample size was, frankly, just barely large enough to maybe provide suggestions for future research. If a meaningful drop-out rate is expected, I would argue that the researchers running these sorts of studies are mostly wasting everyone's time and money for the sake of publishing a few papers.

flatline 8y ago

There is nothing more frustrating than doing things you are bad at, regularly, especially if you are bound to get somewhat worse as time goes on. To your point, even if that were the reason for a significant number of departures, it would not go against the conclusion.

ido 8y ago
Sounds like a use case ripe for game developers to jump on to.

joe_the_user 8y ago

You can't really criticize it only based on those numbers, at least not without looking at similar numbers from other cohort studies with the same length. Well, one can't criticize the quality of the study without looking how well other studies did. But even if all studies involve a massive drop-out rate, the drop-out rate still seems like something to look at in evaluating the accuracy of the study. I mean my concern, looking from the outside, isn't relative accuracy, but absolute accuracy. Not "is it as good as other studies?" but "it is good enough that I can rely on it?"

mjevans 8y ago
15 -years- is a LONG time for a study. I'm not sure about the dropout rates, but it is a real statistical issue. I'm just not sure there's anything that can reasonably be done about it. You'd have to have something like federally mandated participation / continued participation. That's also very unlikely to pass ethics boards. BTW, it would also be an issue if the subjects were compensated to the point that they differed from the actual population at large. Fundamentally having more respect for doing things right, the right way, over the /long/ term, including the STEM fields, would be a 'good thing' for any society, but particularly western ones (where often the bad/evil antagonist is based on science or a cold lack of emotions; but the protagonist is based mostly on emotion and feeling instead of logic).
- joe_the_user 8y ago
  15 -years- is a LONG time for a study. I'm not sure about the dropout rates, but it is a real statistical issue. I'm just not sure there's anything that can reasonably be done about it. Everyone keeps replying as if my comments imply a demand that "something be done about this". That's not my point. Maybe nothing can be done about it. The situation is still significant or at least appears significant.
  - b_tterc_p 8y ago
    Similarly, it may be that running studies like this is really difficult and that massively reduced sample sizes are something we must accept for practical reasons... but that doesn’t mean it permits us to ignore statistical ramifications
digitalzombie 8y ago
> "it is good enough that I can rely on it?" It's called longitudinal analysis or survival analysis, depending on the question being asked. We statistician got test and models for these. You can rely on it if the people who are conducting the experiment are competent and know what they're doing.

nolemurs 8y ago

> This is a cohort study conducted over 15 years. You can't really criticize it only based on those numbers You absolutely can. Scientific studies should be judged on their potential to produce interesting results. This study has too small a sample size to provide results that merit making even the weakest claims about. Even the initial sample size was, frankly, just barely large enough to maybe provide suggestions for future research. If a meaningful drop-out rate is expected, I would argue that the researchers running these sorts of studies are mostly wasting everyone's time and money for the sake of publishing a few papers.