5 ms·

I'm not sure. Psychology and sociology has such a horrible track record when it comes to reproducibility of studies that have some kind of political significanc

I'm not sure. Psychology and sociology has such a horrible track record when it comes to reproducibility of studies that have some kind of political significance that we just have to be careful. In case of these topics, there's a relatively strong publication pressure here. Also: the raw data is unfortunately not enough without pre-registration. (For obvious reasons.) "bad faith" is absolutely required in science. Especially in sociology and psychology, unfortunately.

x86_64Ubuntu 6y ago
I'm not sure there is a crisis of reproducibility in a study like this. We can tell with accuracy what the race of the physician is, the race of the baby and whether or not the baby dies or not.
- AzzieElbab 6y ago
  We also can’t tell what is the death rate of all babies delivered by black doctors from this study
  - x86_64Ubuntu 6y ago
    What does that have to do with anything?
    - AzzieElbab 6y ago
      Maybe they picked up outstanding doctors who deliver outstanding results regardless of patients race? I mean the dataset is so small, one off can derail any conclusion
      - croissants 6y ago
        > I mean the dataset is so small, one off can derail any conclusion What do you mean? Per the Supplement, their data comes from a pool of ~4k white doctors and ~500 black doctors, with hundreds of thousands of observations/patients. This dataset doesn't seem "so small"? [1] https://www.pnas.org/content/pnas/suppl/2020/08/12/1913405117.DCSupplemental/pnas.1913405117.sapp.pdf https://www.pnas.org/content/pnas/suppl/2020/08/12/191340511...
        sz4kerto 6y ago
        It really all depends on the methodology. I recommend some reading about 'p-hacking'. It is not that hard to find to find really shocking findings in large datasets -- the question is how hard you look. In this particular case, the question is how many hypotheses have been evaluated, i.e. has the author 'mined the data' for correlations. If yes, then the acceptable p value is much smaller than normal.
        croissants 6y ago
        I think it's a bit of a reach to suggest that this is p-hacking. Some of the relevant authors have studied this hypothesis (a gap in pregnancy-related outcomes between black and white people) across multiple papers, and the effect found here -- ignoring any p calculation -- is pretty big. It's not like they're picking some extremely specific combination of factors.
        x86_64Ubuntu 6y ago
        And this is the narrative that I'm speaking of when I mention how these discussions go. It's a gish-gallop of saying that since we have imperfect knowledge, we can't draw a single conclusion about phenomena and systems that are well documented. First it's the lack of studies, then you get a study and it's a hidden bias in the populations used in the study, then it's the "low sample size" (that's a favorite), after that it's p-hacking or whatever else. And it goes on and on, even though for hundreds of years, the country and society were tuned to ensure that the black community enjoyed degraded outcomes in all aspects of life.
        deleted 6y ago
        [deleted]
        AzzieElbab 6y ago
        Thanks for posting this, I must have misread the original article. The samples are x10 larger than I understood at first. However, please keep in mind that the subset of white ph/patients is almost 4 times larger than black ph/black patient. Data fitting seem questionable from there on. 4x is not that large of a scale but still analyses should have been more nuanced from here. For example, hospitals are not considered a variable at all. I would think that it is a major one

x86_64Ubuntu 6y ago

I'm not sure there is a crisis of reproducibility in a study like this. We can tell with accuracy what the race of the physician is, the race of the baby and whether or not the baby dies or not.

AzzieElbab 6y ago
We also can’t tell what is the death rate of all babies delivered by black doctors from this study
- x86_64Ubuntu 6y ago
  What does that have to do with anything?
  - AzzieElbab 6y ago
    Maybe they picked up outstanding doctors who deliver outstanding results regardless of patients race? I mean the dataset is so small, one off can derail any conclusion
    - croissants 6y ago
      > I mean the dataset is so small, one off can derail any conclusion What do you mean? Per the Supplement, their data comes from a pool of ~4k white doctors and ~500 black doctors, with hundreds of thousands of observations/patients. This dataset doesn't seem "so small"? [1] https://www.pnas.org/content/pnas/suppl/2020/08/12/1913405117.DCSupplemental/pnas.1913405117.sapp.pdf https://www.pnas.org/content/pnas/suppl/2020/08/12/191340511...
      - sz4kerto 6y ago
        It really all depends on the methodology. I recommend some reading about 'p-hacking'. It is not that hard to find to find really shocking findings in large datasets -- the question is how hard you look. In this particular case, the question is how many hypotheses have been evaluated, i.e. has the author 'mined the data' for correlations. If yes, then the acceptable p value is much smaller than normal.
        croissants 6y ago
        I think it's a bit of a reach to suggest that this is p-hacking. Some of the relevant authors have studied this hypothesis (a gap in pregnancy-related outcomes between black and white people) across multiple papers, and the effect found here -- ignoring any p calculation -- is pretty big. It's not like they're picking some extremely specific combination of factors.
        x86_64Ubuntu 6y ago
        And this is the narrative that I'm speaking of when I mention how these discussions go. It's a gish-gallop of saying that since we have imperfect knowledge, we can't draw a single conclusion about phenomena and systems that are well documented. First it's the lack of studies, then you get a study and it's a hidden bias in the populations used in the study, then it's the "low sample size" (that's a favorite), after that it's p-hacking or whatever else. And it goes on and on, even though for hundreds of years, the country and society were tuned to ensure that the black community enjoyed degraded outcomes in all aspects of life.
      - deleted 6y ago
        [deleted]
      - AzzieElbab 6y ago
        Thanks for posting this, I must have misread the original article. The samples are x10 larger than I understood at first. However, please keep in mind that the subset of white ph/patients is almost 4 times larger than black ph/black patient. Data fitting seem questionable from there on. 4x is not that large of a scale but still analyses should have been more nuanced from here. For example, hospitals are not considered a variable at all. I would think that it is a major one