4 ms·

Let's pick this apart: > This is true, but it's true for almost every observational study, and IMO too easy a dismission. It's also a correct dismissal. The v

by wegs 6y ago

Let's pick this apart: > This is true, but it's true for almost every observational study, and IMO too easy a dismission. It's also a correct dismissal. The vast majority of results don't replicate. It's super-easy to come up with results like this paper just by playing around with what you control for (with a 5% of a false positive each time) until you get the result you want. It's almost impossible to tell if this happened. There are many similar issues. > You can use this reasoning to dismiss, or more likely, to cherry-pick probably 95% of all scientific results from economic/medical/social/nutricion/climate science. And you should! You should believe results only once they've been replicated multiple times, by multiple communities, with multiple methodologies. > I'd say the onus on sceptics in this case The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. > This paper is likely headed for a serious economic journal, where it typically goes through at least days of labor to scrutinise the execution/details/methodology in peer review. I'm not invested enough to volunteer for that. Have you ever gone through a peer review process? Seriously? I understand you're not invested enough to do that, but neither are the reviewers. It's a volunteer position with no credit or upside. Most give a cursory skim, and either accept or reject. It's no better than chance. Seriously. NIPS did a study, and found it is literally no better than chance. Of my own papers, I've only had one where peer reviewers did an in-depth read. For the rest, acceptance was completely random. Reviewer comments often have little relation to the actual paper (many reviewers just skim the abstract and the intro, and don't even fully read those).

em500 6y ago
> The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. The authors posit plausible evidence (observed variance across states with different introduction dates of laws, car ownership, going from 2 to 3 children) that car seat laws was a deterrent to having more than 2 children. A substantial criticism would be to come up with some possible confounders or alternative model of the variation. Saying that it might be caused by anything is a low effort dismissal. > Have you ever gone through a peer review process? Seriously? Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.)
- wegs 6y ago
  > Saying that it might be caused by anything is a low effort dismissal. I am dismissing it as: One paper = interesting hypothesis Multiple replicated papers = theory Multiple replicated papers, with substantially different methodologies = scientific fact > Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.) Interesting. I've worked in several fields, and if this is the case, economics is pretty special. Good on you! It wasn't always this way. As an economist, you study incentives. Think about how to prevent your field from sliding the way others did. Right now, academic incentive structures dictate cursory skim, and cultures+behaviors tend to (often after many years) align to incentives. As someone in a field (apparently) further along that path than economics, all I can suggest is to act sooner rather than later. It's easier to preserve cultures of integrity than to fix cultures of the opposite.
  - em500 6y ago
    I left academics more than 10 years ago. Much easier to earn a good living in industry churning garbage "conversion uplifts" and "data driven insights" and occasionally sounding smart on internet forums with low effort posts than toiling for days/weeks on peer reviews or months/years on publishing. Lecture me again about incentives :)
lootsauce 6y ago
Amen to all of that. This conversation illustrates the challenges with the reproducibility crisis. There is a fundamental problem in how we do science and that problem is hidden by our everyday experience of how effective science has been. This experience colors our perceptions of almost anything that gets dressed up as serious science, including objectively serious science. The scientific process is idealized as this effective tool but research happens at the behest of imperfect institutions and fallible people. Institutions and people are driven by financial and reputational imperative as well as the ideals of their research goals. These can cut strongly against the interests of performing meaningful research and publishing good science.
davidsolomon 6y ago
At a minimum, one thing that we can hopefully convince you of is that our research methodology as designed is reasonably conceptually sound. If we're wrong in our research design, that's a problem regardless of anything else. But if our identification strategy is at least defensible if actually implemented correctly, that's a good start. Having skeptical priors about research in general, whether incentives, error rates, or whatnot, is totally fair.
- wegs 6y ago
  It is a good start! I look at the first paper as generating a hypothesis. That's the first step in any scientific process. A few replications later, we've got a theory, and then a few more later, a fact. The first papers are as important as the rest of that pipeline. What causes outright harm is: 1) Papers with poor methodology; 2) Authors overselling their results; or just as often 3) Readers / reporters giving too much confidence in preliminary results To be clear, I was calling em500 on #3, not the authors on #1 or #2. On a skim, I don't see obvious problems (but, then, I wouldn't expect to find them; most require a deep read). It's published in an open access venue, which is always a good sign. It also helps that you're from BC, which for the most part, (still) has a pretty good reputation for scientific integrity (there's been a rapid decline in scientific integrity over the past two decades, and that's hit the institutional culture across the river from you particularly hard). So you're okay by me!

by wegs 6y ago

em500 6y ago

> The onus is on the people presenting the result, in all cases. That's the scientific process. The authors need to defend the null hypothesis as vigorously as you can. The authors posit plausible evidence (observed variance across states with different introduction dates of laws, car ownership, going from 2 to 3 children) that car seat laws was a deterrent to having more than 2 children. A substantial criticism would be to come up with some possible confounders or alternative model of the variation. Saying that it might be caused by anything is a low effort dismissal. > Have you ever gone through a peer review process? Seriously? Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.)

wegs 6y ago
> Saying that it might be caused by anything is a low effort dismissal. I am dismissing it as: One paper = interesting hypothesis Multiple replicated papers = theory Multiple replicated papers, with substantially different methodologies = scientific fact > Yes I have, on both sides. Maybe it differs by field, but I (and my fellow PhD students) took it serious enough that we rarely read and finished writing a peer review within a single day. (This was mostly on economic theory / methodology papers, where even understanding the paper can take multiple days.) Interesting. I've worked in several fields, and if this is the case, economics is pretty special. Good on you! It wasn't always this way. As an economist, you study incentives. Think about how to prevent your field from sliding the way others did. Right now, academic incentive structures dictate cursory skim, and cultures+behaviors tend to (often after many years) align to incentives. As someone in a field (apparently) further along that path than economics, all I can suggest is to act sooner rather than later. It's easier to preserve cultures of integrity than to fix cultures of the opposite.
- em500 6y ago
  I left academics more than 10 years ago. Much easier to earn a good living in industry churning garbage "conversion uplifts" and "data driven insights" and occasionally sounding smart on internet forums with low effort posts than toiling for days/weeks on peer reviews or months/years on publishing. Lecture me again about incentives :)

lootsauce 6y ago

Amen to all of that. This conversation illustrates the challenges with the reproducibility crisis. There is a fundamental problem in how we do science and that problem is hidden by our everyday experience of how effective science has been. This experience colors our perceptions of almost anything that gets dressed up as serious science, including objectively serious science. The scientific process is idealized as this effective tool but research happens at the behest of imperfect institutions and fallible people. Institutions and people are driven by financial and reputational imperative as well as the ideals of their research goals. These can cut strongly against the interests of performing meaningful research and publishing good science.

davidsolomon 6y ago

At a minimum, one thing that we can hopefully convince you of is that our research methodology as designed is reasonably conceptually sound. If we're wrong in our research design, that's a problem regardless of anything else. But if our identification strategy is at least defensible if actually implemented correctly, that's a good start. Having skeptical priors about research in general, whether incentives, error rates, or whatnot, is totally fair.

wegs 6y ago
It is a good start! I look at the first paper as generating a hypothesis. That's the first step in any scientific process. A few replications later, we've got a theory, and then a few more later, a fact. The first papers are as important as the rest of that pipeline. What causes outright harm is: 1) Papers with poor methodology; 2) Authors overselling their results; or just as often 3) Readers / reporters giving too much confidence in preliminary results To be clear, I was calling em500 on #3, not the authors on #1 or #2. On a skim, I don't see obvious problems (but, then, I wouldn't expect to find them; most require a deep read). It's published in an open access venue, which is always a good sign. It also helps that you're from BC, which for the most part, (still) has a pretty good reputation for scientific integrity (there's been a rapid decline in scientific integrity over the past two decades, and that's hit the institutional culture across the river from you particularly hard). So you're okay by me!