2 ms·

Given that it seems like they analysed real-world data rather than producing synthetic experiments, I wouldn't put too much stock in the findings. For instance,

by 4bpp 11y ago

Given that it seems like they analysed real-world data rather than producing synthetic experiments, I wouldn't put too much stock in the findings. For instance, in a hypothetical world where all businesses contributing to Github projects are perfectly sexist, only ever assign male coders to core engineering and only ever assign female coders to UX design, UX design is intrinsically more controversial and Github maintainerers accept patches purely on the degree of public support for their contents, one would expect a far lower degree of acceptance for patches from identifiable (as businesses presumably make their workers operate under real-name accounts) female accounts without this containing any signal about sexism on the Github side. A better methodology for a study like this would be to identify a number of people planning to make sporadic contributions to projects they have no connection to and making them submit the pull requests under fresh fictional profiles with randomised gender. This way, any irrelevant correlations between the kind of pull request being made and the requesting party being identifiable as female would be eliminated.

nsfyn55 11y ago
I would tend to agree with you. When you use public data you get all the implied imperfections of public data. You can make some vague inferences but since the controls aren't in laboratory conditions the potential for biased results is too high. If they showed individuals with already high acceptance rates received lower acceptance rates on the same code when submitted from a gender identifiable account then I'd find it to be more compelling.
johnhess 11y ago
As the article itself states: > Experiments and retrospective field studies each have advantages. The advantage of experiments is that they can more confidently infer cause and effect by isolating gender as the predictor variable. The advantage of retrospective field studies is that they tend to have higher ecological validity because they are conducted in real-world situations. In this paper, we use a retrospective field study as a first step to quantify the effect of gender bias in open source. The research is presented in context, an important and valid context, too. At the end, they discuss potential explanations and never go so far as to say "And all of this is clearly because ______". A controlled study would be a great followup, but I'd sure put stock in these compelling and well documented findings.

by 4bpp 11y ago

nsfyn55 11y ago

I would tend to agree with you. When you use public data you get all the implied imperfections of public data. You can make some vague inferences but since the controls aren't in laboratory conditions the potential for biased results is too high. If they showed individuals with already high acceptance rates received lower acceptance rates on the same code when submitted from a gender identifiable account then I'd find it to be more compelling.

johnhess 11y ago

As the article itself states: > Experiments and retrospective field studies each have advantages. The advantage of experiments is that they can more confidently infer cause and effect by isolating gender as the predictor variable. The advantage of retrospective field studies is that they tend to have higher ecological validity because they are conducted in real-world situations. In this paper, we use a retrospective field study as a first step to quantify the effect of gender bias in open source. The research is presented in context, an important and valid context, too. At the end, they discuss potential explanations and never go so far as to say "And all of this is clearly because ______". A controlled study would be a great followup, but I'd sure put stock in these compelling and well documented findings.