4 ms·

> It seems like they're taking a bucketload of demographic information and then trying to control for every other possible factor that could affect fertility ra

by em500 6y ago

> It seems like they're taking a bucketload of demographic information and then trying to control for every other possible factor that could affect fertility rates to isolate the effect of car seat laws. I'm skeptical that it's even possible to control for all of those variables. Even assuming that they perfectly model the factors they do account for, it's perfectly possible that there are more underlying factors. This is true, but it's true for almost every observational study, and IMO too easy a dismission. You can use this reasoning to dismiss, or more likely, to cherry-pick probably 95% of all scientific results from economic/medical/social/nutricion/climate science. Just imagine the actual mechanics of setting up a proper Randomized Controlled Trial of seat belt safety! Back to the actual study, the reasoning appearch to hinge on the variation of outcomes between 1) having a 2nd and 3rd child 2) between car and non-car owning families 3) accross different states in the US with different introduction dates between car seat laws. Of course, their might still be lurking confounders, but the presumption that this is really caused by car seat laws sounds quite convincing to me[1]. I'd say the onus on sceptics in this case is to actually propose some possible confounders that might explain variation in all of 1), 2) and 3). [1] Disclaimer: I haven't gone through the details of the paper to see if the idea is actually well executed. This paper is likely headed for a serious economic journal, where it typically goes through at least days of labor to scrutinise the execution/details/methodology in peer review. I'm not invested enough to volunteer for that.

ema 6y ago
> You can use this reasoning to dismiss [...] probably 95% of all scientific results from economic/medical/social/nutricion/climate science. Most researchers are incentived to churn out as many papers as possible. Most people who are making research grant decisions have little incentive to give grants to substantial science over insubstantial science that merely looks good. After all they're unlikely to face career consequences for funding science which doesn't replicate. Many of the people involved in the production of science are disincentivezed to point out problems with the process of science because science funding depends on science being highly esteemed, which only indirectly and with significant time lag depends on the effectiveness of science. All this leads me to the conclusion that, yes, it is entirely possible that 95% of science in those fields (and maybe others) is junk.
- em500 6y ago
  > All this leads me to the conclusion that, yes, it is entirely possible that 95% of science in those fields (and maybe others) is junk. I'm don't think that's farfetched at all. The problem though is that people will not apply uniform 95% skepticism to all studies. They'll just embrace or dismiss them based how how aligned the results are with preconceptions.
avip 6y ago
>Just imagine the actual mechanics of setting up a proper Randomized Controlled Trial of seat belt safety! https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094
- freetime2 6y ago
  I laughed out loud when I got to this part: > Intervention: Jumping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack (unblinded).
  - gpderetta 6y ago
    also: "the trial was only able to enroll participants on small stationary aircraft on the ground, suggesting cautious extrapolation to high altitude jumps." :) In realty I think a proper trial is supposed to always use the best known treatment as control.
wegs 6y ago
Let's pick this apart: > This is true, but it's true for almost every observational study, and IMO too easy a dismission. It's also a correct dismissal. The vast majority of results don't replicate. It's super-easy to come up with results like this paper just by playing around with what you control for (with a 5% of a false positive each time) until you get the result you want. It's almost impossible to tell if this happened. There are many similar issues. > You can use this reasoning to dismiss, or more likely, to cherry-pick probably 95% of all scientific results from economic/medical/social/nutricion/climate science. And you should! You should believe results only once they've been replicated multiple times, by multiple communities, with multiple methodologies. > I'd say the onus on sceptics in this case The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. > This paper is likely headed for a serious economic journal, where it typically goes through at least days of labor to scrutinise the execution/details/methodology in peer review. I'm not invested enough to volunteer for that. Have you ever gone through a peer review process? Seriously? I understand you're not invested enough to do that, but neither are the reviewers. It's a volunteer position with no credit or upside. Most give a cursory skim, and either accept or reject. It's no better than chance. Seriously. NIPS did a study, and found it is literally no better than chance. Of my own papers, I've only had one where peer reviewers did an in-depth read. For the rest, acceptance was completely random. Reviewer comments often have little relation to the actual paper (many reviewers just skim the abstract and the intro, and don't even fully read those).
- em500 6y ago
  > The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. The authors posit plausible evidence (observed variance across states with different introduction dates of laws, car ownership, going from 2 to 3 children) that car seat laws was a deterrent to having more than 2 children. A substantial criticism would be to come up with some possible confounders or alternative model of the variation. Saying that it might be caused by anything is a low effort dismissal. > Have you ever gone through a peer review process? Seriously? Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.)

by em500 6y ago

ema 6y ago

> You can use this reasoning to dismiss [...] probably 95% of all scientific results from economic/medical/social/nutricion/climate science. Most researchers are incentived to churn out as many papers as possible. Most people who are making research grant decisions have little incentive to give grants to substantial science over insubstantial science that merely looks good. After all they're unlikely to face career consequences for funding science which doesn't replicate. Many of the people involved in the production of science are disincentivezed to point out problems with the process of science because science funding depends on science being highly esteemed, which only indirectly and with significant time lag depends on the effectiveness of science. All this leads me to the conclusion that, yes, it is entirely possible that 95% of science in those fields (and maybe others) is junk.

em500 6y ago
> All this leads me to the conclusion that, yes, it is entirely possible that 95% of science in those fields (and maybe others) is junk. I'm don't think that's farfetched at all. The problem though is that people will not apply uniform 95% skepticism to all studies. They'll just embrace or dismiss them based how how aligned the results are with preconceptions.

avip 6y ago

>Just imagine the actual mechanics of setting up a proper Randomized Controlled Trial of seat belt safety! https://www.bmj.com/content/363/bmj.k5094 https://www.bmj.com/content/363/bmj.k5094

freetime2 6y ago
I laughed out loud when I got to this part: > Intervention: Jumping from an aircraft (airplane or helicopter) with a parachute versus an empty backpack (unblinded).
- gpderetta 6y ago
  also: "the trial was only able to enroll participants on small stationary aircraft on the ground, suggesting cautious extrapolation to high altitude jumps." :) In realty I think a proper trial is supposed to always use the best known treatment as control.

wegs 6y ago

Let's pick this apart: > This is true, but it's true for almost every observational study, and IMO too easy a dismission. It's also a correct dismissal. The vast majority of results don't replicate. It's super-easy to come up with results like this paper just by playing around with what you control for (with a 5% of a false positive each time) until you get the result you want. It's almost impossible to tell if this happened. There are many similar issues. > You can use this reasoning to dismiss, or more likely, to cherry-pick probably 95% of all scientific results from economic/medical/social/nutricion/climate science. And you should! You should believe results only once they've been replicated multiple times, by multiple communities, with multiple methodologies. > I'd say the onus on sceptics in this case The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. > This paper is likely headed for a serious economic journal, where it typically goes through at least days of labor to scrutinise the execution/details/methodology in peer review. I'm not invested enough to volunteer for that. Have you ever gone through a peer review process? Seriously? I understand you're not invested enough to do that, but neither are the reviewers. It's a volunteer position with no credit or upside. Most give a cursory skim, and either accept or reject. It's no better than chance. Seriously. NIPS did a study, and found it is literally no better than chance. Of my own papers, I've only had one where peer reviewers did an in-depth read. For the rest, acceptance was completely random. Reviewer comments often have little relation to the actual paper (many reviewers just skim the abstract and the intro, and don't even fully read those).

em500 6y ago
> The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. The authors posit plausible evidence (observed variance across states with different introduction dates of laws, car ownership, going from 2 to 3 children) that car seat laws was a deterrent to having more than 2 children. A substantial criticism would be to come up with some possible confounders or alternative model of the variation. Saying that it might be caused by anything is a low effort dismissal. > Have you ever gone through a peer review process? Seriously? Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.)