3 ms·

So, I agree it's an interesting idea, and I think this small study could hopefully spur more conclusive ones, but... 1) N=45 2) based on "semi-structured phon

So, I agree it's an interesting idea, and I think this small study could hopefully spur more conclusive ones, but... 1) N=45 2) based on "semi-structured phone interview" Neither of these is a deal-breaker, but what you'd really prefer is some sort of objective measure of sleep quantity, and how it impacted the person. Teenagers are, by and large, entirely aware that when adults ask them questions, they are likely to draw conclusions about policy towards teenagers. They want to sleep later. How likely is it that they would admit it, if they didn't really get any more sleep out of this? Not saying it's not true, or that it isn't an interesting idea to try to put this "natural experiment" to work somehow, but I'm not sure this is really enough to draw any conclusions, and I actually already agreed with their conclusion before seeing this study. A better method might be to compare home-school students to public schools, or compare summer-break to school year, and to use some more objective metric than a phone interview.

rcar 6y ago
Agreed with all points. I often find that these sorts of observational studies with weak methods that provide evidence for something the unblinded study participants would want to support (e.g., this study, observational studies of the effect of reduced workweek hours on productivity, many of the observational basic income studies) are really hard to evaluate fairly. It becomes easy to latch onto the conclusions when the intervention is one you're predisposed to believe in and to latch onto the methodological weaknesses when it's one you start of opposed towards, and so I feel like they don't really advance science forward at all.
- danmcs 6y ago
  I think it's useful to think of studies like this as a MVP. They're relatively cheap and fast. If they give a null result, you've failed fast. If they give a non-null, then you can invest in further study and iterate.
  - emteycz 6y ago
    As opposed to a MVP, such study can give you any results you want.
- throwaway_pdp09 6y ago
  They provide dirty data but cheaply which can be used to prod a more rigorous study into life. So they do advance science. It isn't all sterile test tubes and lab coats, at the coal face it gets mucky, it has to.
speedgoose 6y ago
It's the old qualitative vs quantitative debate. Interviewing 45 people can provide much more meaningful information than sending a survey to 1000 people. I'm not saying it's the case there, I'm not an expert in this domain. But 45 semi structured phones interviews is perfectly fine for a lot of research.
- emteycz 6y ago
  I made 45 consecutive structured calls in my life, many times actually. I always could interpret the result in many ways, most of the time complete opposites were the leading interpretations. Their actual behavior afterwards told me the reality. People on the phone will say a lot of things, and it's very easy to accidentally make them reply what you want to hear even though they don't mean it.

rcar 6y ago

Agreed with all points. I often find that these sorts of observational studies with weak methods that provide evidence for something the unblinded study participants would want to support (e.g., this study, observational studies of the effect of reduced workweek hours on productivity, many of the observational basic income studies) are really hard to evaluate fairly. It becomes easy to latch onto the conclusions when the intervention is one you're predisposed to believe in and to latch onto the methodological weaknesses when it's one you start of opposed towards, and so I feel like they don't really advance science forward at all.

danmcs 6y ago
I think it's useful to think of studies like this as a MVP. They're relatively cheap and fast. If they give a null result, you've failed fast. If they give a non-null, then you can invest in further study and iterate.
- emteycz 6y ago
  As opposed to a MVP, such study can give you any results you want.
throwaway_pdp09 6y ago
They provide dirty data but cheaply which can be used to prod a more rigorous study into life. So they do advance science. It isn't all sterile test tubes and lab coats, at the coal face it gets mucky, it has to.

speedgoose 6y ago

It's the old qualitative vs quantitative debate. Interviewing 45 people can provide much more meaningful information than sending a survey to 1000 people. I'm not saying it's the case there, I'm not an expert in this domain. But 45 semi structured phones interviews is perfectly fine for a lot of research.

emteycz 6y ago
I made 45 consecutive structured calls in my life, many times actually. I always could interpret the result in many ways, most of the time complete opposites were the leading interpretations. Their actual behavior afterwards told me the reality. People on the phone will say a lot of things, and it's very easy to accidentally make them reply what you want to hear even though they don't mean it.