5 ms·

Please support your argument with numbers. 36 participants can be plenty or can be absolutely insignificant. My partner is just in the process of getting subjec

Please support your argument with numbers. 36 participants can be plenty or can be absolutely insignificant. My partner is just in the process of getting subjects for an experiment and she's trying to estimate the required number of participants. In their case 30 participants would make the experiment useless, 40-42 would make it very powerful (significant). You can't judge the weakness of an experiment just based on the number of participants. Your comment might mislead others with not enough knowledge of stats I think.

c1as 8y ago
Academia has had their chance. No one believes "studies" about personal life choices any more if the results conflict with each other every 5 years. It's just publication pressure (or worse, as other commenters have hinted at).
- sz4kerto 8y ago
  > Academia has had their chance. You can't say that. (Ok, of course everyone has the right to her opinions.) The point of the scientific method is that it is, in many ways, self-correcting. It's getting addressed, albeit slowly. It will be counterproductive for all of us if we simply dismiss all results we're not comfortable with that it's only publication pressure. I mean -- you can say that personally, and it probably makes sense, but as a society we need to get social sciences and medicine into a better shape instead of just 'letting it go'.
  - pelasaco 8y ago
    > It will be counterproductive for all of us if we simply dismiss all results we're not comfortable with that it's only publication pressure. That's not what he said. Just those which "each 5 years" have completely different results.
- Ragnarork 8y ago
  > Academia has had their chance Oh, so what should we go to now in order to try to expand our knowledge about this? What are you suggesting?
  - Frondo 8y ago
    This is always the implicit question, and I'm guessing the answer is scientism -- material that has the form of academic product, but isn't academic. Like Jordon Peterson videos or a lot of the scientific racism stuff on Youtube. It's draped in the language of knowing, but it's purely emotional material; stuff that makes you feel good, and makes you feel smart for feeling right. (I also think the desire to drape our tribal impulses in scientific form comes from society prizing smarts for so very, very long, but that's another story.)
    - unabst 8y ago
      Jordan Peterson is as scientific as a social scientist gets. He cites papers and describes the work of past scientists and his premises are evidence driven. He is a professor; a professional academic. He has his opinions, positions and motives just like anyone else, but is open about them. If anything he is a rock star lecturer that exemplifies how informative, accurate, sincere, progressive, intimate, and inspiring science can be. He helps people for a living as a psychologist and a lecturer. If he offends you then you either have issues or don't understand him or both, but he would gladly engage with you either way. Most that disagree with him disagree with the science, which puts them in the same boat as climate change deniers, of which there are plenty voicing their alternate facts online. But their voice doesn't validate their claims. Science is not a democracy.
      - gboudrias 8y ago
        > If he offends you then you either have issues or don't understand him or both, but he would gladly engage with you either way. Hmmm no we can also think he uses mostly weak papers with dodgy methodologies and over-generalizes the conclusions. Not all scientists within a field agree with any given scientist of that field, that would be ludicrous.
        unabst 8y ago
        How do you know he uses mostly weak papers? That is an overgeneralization. And why would he? Which papers should he have chosen? And I never said all scientists agree. Typical straw man. But what I will say is there are plenty of climate change deniers that claim to be scientists.
- RankingMember 8y ago
  You're placing the blame in the wrong places, lumping in the above-board journals for the actions of those who aren't. If you want to blame someone, blame the trashier elements of our media, who jump all over the constant back and forth "studies" published in pay-to-play junk journals as if they're handed down from a great deity on stone tablets.
alexpetralia 8y ago
This is an important counterargument. Effect size absolutely matters.
- Bartweiss 8y ago
  For that matter, very small studies are safer from certain types of false positive. The file-drawer problem with small lab studies is serious, and can't be solved with effect sizes; it's much easier to run a few of these and publish the hits than to do that with large-population studies. But small lab studies provide a lot of value alongside population studies. They help reveal issues with bad self-report data, which are major concerns for sleep and weight questions. They also offer an efficient way to replicate and extend large studies. If you're spending a decade with 10k+ subjects, gathering a huge range of data is understandable (and I would argue, preferable). But it leaves you with a choice between salami slicing across all your variables, or applying multiple-comparisons corrections that destroy your sensitivity. Using large multi-factor studies as exploratory work, then validating effects with small focused studies is actually a pretty healthy outcome. A final thought: I have a lot of problems with this study, but file-drawer issues aren't one of them. Looking at a bunch of correlated variables (insulin sensitivity, weight gain, outside-meal eating, etc) and reporting them all is a solid way to mitigate that - we know those things should correlate for basically any real effect, which makes it much harder to stumble into a false positive and publish.
deleted 8y ago
[deleted]
5555624 8y ago
The fourth paragraph of the article links to an article titled, "Sleeping in on the weekends can compensate for lack of sleep during the week, study suggests." (https://www.washingtonpost.com/news/speaking-of-science/wp/2018/05/23/people-who-sleep-in-on-weekends-avoid-dying-young-study-suggests/?utm_term=.283730348877 https://www.washingtonpost.com/news/speaking-of-science/wp/2...) That study tracked 38,000 people over 13 years. "People who slept for fewer than the recommended seven hours each weekday, but caught an extra hour or two on weekends, lived just as long as people who always slept seven hours, the authors reported."
Bartweiss 8y ago
36 participants is reasonable for showing some effects, and it is easy for people who've just internalized "small samples bad" to underestimate that. I'm glad you raised the point, especially since ignoring it tends to push people to respect very large studies - even when those studies use self-report on famously unreliable topics like weight and sleep. I still find it worrying here because we know there's substantial variance in how/when people sleep, and 36 is well below "representative sample" numbers. More generally, a 36 participant, 2 week study ought to be examined pretty carefully when it contradicts a study (Akerstedt et al) on 38,000 subjects over 13 years. I can hardly blame the authors for that, though; small lab confirmations of large self-report studies are a useful and established practice. But I don't think that's the damning part. The 9 night study span is incredibly worrying for trying to assess a long-term effect. If a value shifts over that span, is it rising steadily, rising to a new equilibrium, or reacting to a change and then returning to homeostasis? (I do want to give credit to the study for using pre-experiment substance/diet/sleep control, and then applying individual sleep times to avoid "night owls" being a confounder. Seeing studies judge "sleep time" by making all subjects go to bed at the same hour is infuriating.) Finally, there's strong evidence that something was unusual about study conditions. The WaPo article claims that the weekend sleepers "gained nearly three pounds over two weeks", but that's pretty much journalistic malpractice. The study actually found that all patients in the study gained weight; more weight with higher confidence for the low-sleep groups, but the groups had heavy overlap in amount gained. We can be confident that two weeks of work-schedule sleep do not cause that sort of weight gain for most people, since the reduction to the absurd there is everyone gaining ~70 lb/year indefinitely. It's the sort of number which raises pretty significant questions about how this compares to anything outside the lab setting.

c1as 8y ago

Academia has had their chance. No one believes "studies" about personal life choices any more if the results conflict with each other every 5 years. It's just publication pressure (or worse, as other commenters have hinted at).

sz4kerto 8y ago
> Academia has had their chance. You can't say that. (Ok, of course everyone has the right to her opinions.) The point of the scientific method is that it is, in many ways, self-correcting. It's getting addressed, albeit slowly. It will be counterproductive for all of us if we simply dismiss all results we're not comfortable with that it's only publication pressure. I mean -- you can say that personally, and it probably makes sense, but as a society we need to get social sciences and medicine into a better shape instead of just 'letting it go'.
- pelasaco 8y ago
  > It will be counterproductive for all of us if we simply dismiss all results we're not comfortable with that it's only publication pressure. That's not what he said. Just those which "each 5 years" have completely different results.
Ragnarork 8y ago
> Academia has had their chance Oh, so what should we go to now in order to try to expand our knowledge about this? What are you suggesting?
- Frondo 8y ago
  This is always the implicit question, and I'm guessing the answer is scientism -- material that has the form of academic product, but isn't academic. Like Jordon Peterson videos or a lot of the scientific racism stuff on Youtube. It's draped in the language of knowing, but it's purely emotional material; stuff that makes you feel good, and makes you feel smart for feeling right. (I also think the desire to drape our tribal impulses in scientific form comes from society prizing smarts for so very, very long, but that's another story.)
  - unabst 8y ago
    Jordan Peterson is as scientific as a social scientist gets. He cites papers and describes the work of past scientists and his premises are evidence driven. He is a professor; a professional academic. He has his opinions, positions and motives just like anyone else, but is open about them. If anything he is a rock star lecturer that exemplifies how informative, accurate, sincere, progressive, intimate, and inspiring science can be. He helps people for a living as a psychologist and a lecturer. If he offends you then you either have issues or don't understand him or both, but he would gladly engage with you either way. Most that disagree with him disagree with the science, which puts them in the same boat as climate change deniers, of which there are plenty voicing their alternate facts online. But their voice doesn't validate their claims. Science is not a democracy.
    - gboudrias 8y ago
      > If he offends you then you either have issues or don't understand him or both, but he would gladly engage with you either way. Hmmm no we can also think he uses mostly weak papers with dodgy methodologies and over-generalizes the conclusions. Not all scientists within a field agree with any given scientist of that field, that would be ludicrous.
      - unabst 8y ago
        How do you know he uses mostly weak papers? That is an overgeneralization. And why would he? Which papers should he have chosen? And I never said all scientists agree. Typical straw man. But what I will say is there are plenty of climate change deniers that claim to be scientists.
RankingMember 8y ago
You're placing the blame in the wrong places, lumping in the above-board journals for the actions of those who aren't. If you want to blame someone, blame the trashier elements of our media, who jump all over the constant back and forth "studies" published in pay-to-play junk journals as if they're handed down from a great deity on stone tablets.

alexpetralia 8y ago

This is an important counterargument. Effect size absolutely matters.

Bartweiss 8y ago
For that matter, very small studies are safer from certain types of false positive. The file-drawer problem with small lab studies is serious, and can't be solved with effect sizes; it's much easier to run a few of these and publish the hits than to do that with large-population studies. But small lab studies provide a lot of value alongside population studies. They help reveal issues with bad self-report data, which are major concerns for sleep and weight questions. They also offer an efficient way to replicate and extend large studies. If you're spending a decade with 10k+ subjects, gathering a huge range of data is understandable (and I would argue, preferable). But it leaves you with a choice between salami slicing across all your variables, or applying multiple-comparisons corrections that destroy your sensitivity. Using large multi-factor studies as exploratory work, then validating effects with small focused studies is actually a pretty healthy outcome. A final thought: I have a lot of problems with this study, but file-drawer issues aren't one of them. Looking at a bunch of correlated variables (insulin sensitivity, weight gain, outside-meal eating, etc) and reporting them all is a solid way to mitigate that - we know those things should correlate for basically any real effect, which makes it much harder to stumble into a false positive and publish.

deleted 8y ago

[deleted]

5555624 8y ago

The fourth paragraph of the article links to an article titled, "Sleeping in on the weekends can compensate for lack of sleep during the week, study suggests." (https://www.washingtonpost.com/news/speaking-of-science/wp/2018/05/23/people-who-sleep-in-on-weekends-avoid-dying-young-study-suggests/?utm_term=.283730348877 https://www.washingtonpost.com/news/speaking-of-science/wp/2...) That study tracked 38,000 people over 13 years. "People who slept for fewer than the recommended seven hours each weekday, but caught an extra hour or two on weekends, lived just as long as people who always slept seven hours, the authors reported."

Bartweiss 8y ago

36 participants is reasonable for showing some effects, and it is easy for people who've just internalized "small samples bad" to underestimate that. I'm glad you raised the point, especially since ignoring it tends to push people to respect very large studies - even when those studies use self-report on famously unreliable topics like weight and sleep. I still find it worrying here because we know there's substantial variance in how/when people sleep, and 36 is well below "representative sample" numbers. More generally, a 36 participant, 2 week study ought to be examined pretty carefully when it contradicts a study (Akerstedt et al) on 38,000 subjects over 13 years. I can hardly blame the authors for that, though; small lab confirmations of large self-report studies are a useful and established practice. But I don't think that's the damning part. The 9 night study span is incredibly worrying for trying to assess a long-term effect. If a value shifts over that span, is it rising steadily, rising to a new equilibrium, or reacting to a change and then returning to homeostasis? (I do want to give credit to the study for using pre-experiment substance/diet/sleep control, and then applying individual sleep times to avoid "night owls" being a confounder. Seeing studies judge "sleep time" by making all subjects go to bed at the same hour is infuriating.) Finally, there's strong evidence that something was unusual about study conditions. The WaPo article claims that the weekend sleepers "gained nearly three pounds over two weeks", but that's pretty much journalistic malpractice. The study actually found that all patients in the study gained weight; more weight with higher confidence for the low-sleep groups, but the groups had heavy overlap in amount gained. We can be confident that two weeks of work-schedule sleep do not cause that sort of weight gain for most people, since the reduction to the absurd there is everyone gaining ~70 lb/year indefinitely. It's the sort of number which raises pretty significant questions about how this compares to anything outside the lab setting.