19 ms·

Pull request acceptance of women versus men

177 points by schmichael 11y ago

astine 11y ago
We considered a GitHub profile as gender neutral if all of the following conditions were met: - an identicon (rather than a profile image) was used, - the gender inference tool output a ’unknown’ for the user’s login name and display name, and - none of the panelists indicated that they could identify the user’s gender. Across both panels, panelists inspected 3000 profiles of roughly equal numbers of women and men. Given that acceptance rates dropped for both men and women when this method determined that a user was not gender-neutral, I wonder if use of a profile picture or real name is the real killer here. Perhaps the kind of person to include identifying information in their Github profile is less likely to have good pull requests? More likely, perhaps having any identifying information at all gives people an excuse to reject your pull requests? That would explain why there was a drop for both sexes but a slightly stronger one for women. If you are identifiable from your profile image or name, people are more likely to reject your PR on the bases of your ethnicity, appearance, religion, or gender, with gender being only a part of the effect.
- kuschku 11y ago
  > Perhaps the kind of person to include identifying information in their Github profile is less likely to have good pull requests? Probably commercial employees?
zamalek 11y ago
Open Closed Merged Merge Rate 95% Confidence Interval Women 8,216 21,890 111,011 78.6% [78.45%, 78.87%] Men 150,248 591,785 2,181,517 74.6% [74.56%, 74.67%] > The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men! This difference is statistically significant (p < .001). Many hypothesis are discussed in an attempt to explain this and the paper is well worth the read. Thanks to a documented GitHub API this stuff gets quite scientifically correct - everything is backed up with real-world data. More quotes: > To summarize this paper’s observations: > 1. Women are more likely to have pull requests accepted than men. > 2. Women continue to have high acceptance rates as they gain experience. > 3. Women’s pull requests are less likely to serve an immediate project need. > 4. Women’s changes are larger. > 5. Women’s acceptance rates are higher across programming languages. > 6. Women have lower acceptance rates as outsiders when they are identifiable as women.
- xbryanx 11y ago
  [Edit] Retracting my comment below, after the parent comment was edited to provide more context. --- This quote doesn't seem representative of the whole story that is easily discoverable in the abstract. > Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless.
  - jimrandomh 11y ago
    This is indeed in their abstract, but it's misleading. Take a look at page 15 of the preprint. Their data shows that gendered profiles have lower acceptance rates for both genders; that is, pull requests from a gender-neutral profile are more likely to be accepted regardless of the gender of the person behind it. This paper gives the impression that the authors started out hoping to find evidence of discrimination in the form of a lower pull acceptance rate for women than men, they found the opposite, and then they data mined until they could find something they could use to argue there's an anti-woman bias.
- tyingq 11y ago
  Except, later.... "For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable."
  - nightski 11y ago
    Ignoring the fact that this difference could easily be explained away by some other factor when it is considered. It may have nothing to do with it. Causal relationships are so tricky when dealing with statistics.
  - jjcm 11y ago
    I'm confused - how does that add up to the average of 78.6% then?
tomjen3 11y ago
Can we please try not to make the comments ending up in a giant cluster-fuck, despite how passionate we feel about the subject?
- Analemma_ 11y ago
  You must be new here.
  - booop 11y ago
    Well, his account is 1937 days older than yours.
    - ashi 11y ago
      It is a common expression suggesting one isn't familiar with the ways of the site/community. You know, for humour.
kstrauser 11y ago
I got to > Our results suggest that although women on GitHub may be more competent overall and stopped. No. Just, no. I'm inclined to disbelieve any study that claims one gender is better at programming than other, and that claim makes me automatically dismiss the rest of its conclusions.
- StavrosK 11y ago
  You're being downvoted, but what if the result was "Our results suggest that although men on GitHub may be more competent overall"? Then you'd probably be upvoted. Also, I refuse to believe that genders are obviously different in various physiological and mental traits (e.g. height, spatial awareness, ability to host a baby) except the ones that are socially uncomfortable to us.
  - mucker 11y ago
    No, that is very unlikely.
  - deleted 11y ago
    [deleted]
  - kstrauser 11y ago
    I suspect you're right about the first part. That makes me sad, because I'd hope we'd be more even-keeled as a community. I haven't seen any evidence that the second is true, though. I've not heard of a plausible mechanism by which - all else equal - men would be better at abstract reasoning than women or vice versa.
    - StavrosK 11y ago
      I haven't seen any evidence that it's false, though, either, and there are obviously physical and mental differences between sexes/races/whatever, so it seems more likely to me that this would follow the rule, rather than be the exception.
- mucker 11y ago
mark-ruwt 11y ago
There's a lot in here, and many of the findings will be different than one would expect from the title. If you don't have much time, skip to the "Discussion" section on pages 16 and 17.
mucker 11y ago
The abstract, despite the clear results, is written in the most hostile manner possible.
- jcoffland 11y ago
  You must be easily offended. I did not find it hostile.
  - pluma 11y ago
    I think the proper idiom is "pouring gasoline on the fire". Especially considering the authors say they intentionally cut the analysis "to keep it crisp" when the analysis actually disagreed with the spirit of the abstract.
    - mucker 11y ago
      Exactly. The title intentionally conflicts with the theme.
Sir_Substance 11y ago
Remember when people on the internet werent pushed towards using real names? Gender bias didn't exist because only people you told knew your gender. Such gay times they were!
- pluma 11y ago
  > Such gay times they were! "When men were men, women were men and 12 year old girls were FBI agents"
  - deleted 11y ago
    [deleted]
- jacalata 11y ago
  It was so awesome when you were fine if you either were male or knew enough to pretend to be male!
  - Sir_Substance 11y ago
    If that was how it was for you, you were hanging out in a pretty crummy part of the internet, which is really sad considering how easy it is to find a different part. You missed out on some quality times.
    - jacalata 11y ago
      That is a completely different argument to the one you made originally - "some parts of the internet weren't sexist" vs "people weren't sexist because they couldn't figure out your gender" (which is the one I disagreed with")
- anigbrowl 11y ago
  Gender bias didn't exist Yes, I do remember that (having been using the internet several years before there was any such thing as a web browser), and I dispute your claim, albeit on the purely anecdotal basis of spending years on Usenet.
Rangi42 11y ago
> Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles. I wonder if there's a gender bias with regard to making your Google+ profile public (or having one in the first place).
- protomyth 11y ago
  I get the feeling the peer review is going to flesh some of these issues out. I would love to have time to go over their sample selection.
- groby_b 11y ago
  I also wonder how they actually verified the public G+ profiles. I know that a significant number of women in my circle of acquaintances deliberately flag themselves as male.
  - markus2012 11y ago
    I randomly lie about my gender, age, and anything that could be used to identify me. I learned this from others, and have also encourage my spouse/children/etc. to do the same (which they seem to do). There is no compelling reason for me to provide any real information about myself online.
deleted 11y ago
[deleted]
melted 11y ago
Sigh. Authors are unaware of Simpson's paradox. Such analyses are meaningless if you don't control for such obvious factors as the distributions of projects people of different genders choose to contribute to. See the classic college admissions bias study to understand why: https://en.m.wikipedia.org/wiki/Simpson%27s_paradox https://en.m.wikipedia.org/wiki/Simpson%27s_paradox
- dang 11y ago
  You have a good point, but you've used the format of a middlebrow dismissal to express it—i.e. acting like others don't know something well-known, and snarking about it. That makes it a bad HN comment even if you're right. A better HN comment would bring up the issue neutrally and then either examine the OP to see how it deals with it, or ask a question about how it does.
  - deleted 11y ago
    [deleted]
    - deleted 11y ago
      [deleted]
  - V-2 11y ago
    Grandparent, you should have said something along the lines of "authors didn't give an indication of knowing the Simpson's paradox even though their paper was a great opportunity to do so". Or hear from forum lawyers :)
- minimaxir 11y ago
  Can you give an example of a type of project which would theoetically have different gender distributions of contributors than other projects? That doesn't seem obvious to me.
  - melted 11y ago
    Have no idea. But also have an inkling that distributions would differ (and not necessarily the way you'd expect) in projects that require a specialized skill set, such as e.g. top notch design skills or knowledge of DB internals and such. The point being, as stated the whole thing very much resembles the Berkeley gender discrimination study, so proposed improvements are quite obvious: break things down further by project, see if you can glean something from per-project (or at least per-domain) distribution. There might be nothing there, mind you, and the stated conclusions might still hold, but without controlling for the confounding factor there's no way to tell, and the conclusion of this study is therefore not above suspicion.
sergiotapia 11y ago
>Our analysis (not in this paper -- we've cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women.
cushychicken 11y ago
A good start is to call out misogyny when you see it, online or anywhere else. This is disappointingly rare, in my experience. Edit: what a shotgun blast of downvotes and straw men. People are lining up to say all sorts of utter bullshit. For example: * "Where does this indicate misogyny?" * "The paper is about GitHub, which has a reputation for the opposite." * "A good start is to stop calling things misogyny just because you're bad at reading statistics." * "Of course it's important to recognize when it's actual misogyny and not just disagreement with the code." The paper's findings are clear as a fucking bell: women have equal acceptance of pull requests, except when the pull request obviously comes from a woman. Fucking TEXTBOOK misogyny, and you're an idiotic trog if you don't think so. (Looking RIGHT AT YOU here, pluma.)
- Zikes 11y ago
  However rare it might be, the devastating affects of false positives make this a terrible idea. Just ask Tim Hunt, Matt Taylor, or the men targeted by "donglegate".
  - mjb394 11y ago
    Anti-sexist reactions are very rare compared to instances of sexist behavior in my experience. If anti-sexist behavior was as rampant and casual as sexist behavior, those kind of blowups wouldn't happen.
    - Zikes 11y ago
      There are only two appropriate ways to handle it: if the offense is not egregious, address the incident with the offender in private. If the offense is egregious, or if you do not feel comfortable enough to discuss it with the offender, then you should report it to whatever authority figures are responsible for the context of the incident. "Calling out" the behavior consistently creates vigilantism, and puts an unfair burden on the accused. In instances of sexism I have never, ever seen the accused treated as "innocent until proven guilty" in the eyes of the online hate mobs. Tim Hunt's experience was bad enough that he admitted to thoughts of suicide. This is a man fully respected in his field and has a knighthood, but still had difficulty facing the onslaught of hatred.
noddingham 11y ago
The summary seems to be all positive regarding women's contributions except for the last one: "Women have lower acceptance rates as outsiders when they are identifiable as women.". Looking at the data, they say that the acceptance rate for female outsiders is 62.5%, but they don't say what it is for men. Looking at the chart it looks like ~63.5%. And acceptance rates for both genders drop by >=10% going from non-gendered to gendered. Can we infer then that the bias is somewhat across the board for pull requests from easily gendered users?
- notatoad 11y ago
  it looks like the criteria for determining gender is to pull it from a google+ profile that shares an email address with the github account, and is not using any information from the github profile. It seems likely to me that the bias here is gmail users versus non-gmail (corporate domain?) users.
  - Lawtonfogle 11y ago
    Hmm..., this could very well modify the results. Do they present the numbers when they limit it to only people with google+ accounts or to only people using @gmail as their email address?
  - astine 11y ago
    That's incorrect. The study identified the gender of the Github users twice. They got their 'canonical' gender for users using your method (and removed from their sample users who could not be identified with this method,) but later to figure out whether a user's gender was identifiable, they used the Github user's own profile. Non-Google+ users were not analyzed at all. That could possibly still influence the results, but not the way you are suggesting.
- gregdoesit 11y ago
  It does seem like this would be an even more important conclusion. Whether you're a man or a women, your PR will much more likely be accepted of your profile doesn't reveal your gender (at least not in the sense that creators of this study can identify it).
4bpp 11y ago
Given that it seems like they analysed real-world data rather than producing synthetic experiments, I wouldn't put too much stock in the findings. For instance, in a hypothetical world where all businesses contributing to Github projects are perfectly sexist, only ever assign male coders to core engineering and only ever assign female coders to UX design, UX design is intrinsically more controversial and Github maintainerers accept patches purely on the degree of public support for their contents, one would expect a far lower degree of acceptance for patches from identifiable (as businesses presumably make their workers operate under real-name accounts) female accounts without this containing any signal about sexism on the Github side. A better methodology for a study like this would be to identify a number of people planning to make sporadic contributions to projects they have no connection to and making them submit the pull requests under fresh fictional profiles with randomised gender. This way, any irrelevant correlations between the kind of pull request being made and the requesting party being identifiable as female would be eliminated.
- nsfyn55 11y ago
  I would tend to agree with you. When you use public data you get all the implied imperfections of public data. You can make some vague inferences but since the controls aren't in laboratory conditions the potential for biased results is too high. If they showed individuals with already high acceptance rates received lower acceptance rates on the same code when submitted from a gender identifiable account then I'd find it to be more compelling.
- johnhess 11y ago
  As the article itself states: > Experiments and retrospective field studies each have advantages. The advantage of experiments is that they can more confidently infer cause and effect by isolating gender as the predictor variable. The advantage of retrospective field studies is that they tend to have higher ecological validity because they are conducted in real-world situations. In this paper, we use a retrospective field study as a first step to quantify the effect of gender bias in open source. The research is presented in context, an important and valid context, too. At the end, they discuss potential explanations and never go so far as to say "And all of this is clearly because ______". A controlled study would be a great followup, but I'd sure put stock in these compelling and well documented findings.
BinaryIdiot 11y ago
Remember this is not peered reviewed. Others haven't looked at it to decide if it had proper controls, good enough samples, or even valid data. So take this with a grain of salt until it's properly peered reviewed. I almost wonder if it's worse to publish non-peered-reviewed research or not. I mean it could be great if it's solid research but if there are flaws (even minor) it could undermine other studies, change people's minds, etc.
luso_brazilian 11y ago
Here is one (IMO) much more complete abstract (not to call it tl:dr) that covers the main hypothesis, and all the variables studied Our main research question was: to what extent does gender bias exist among people who judge GitHub pull requests? To answer this question, we approached the problem by examining whether men and women are equally likely to have their pull requests accepted on GitHub, then investigated why differences might exist. We extract users' email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users' profiles. Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles. == Results == Are women's pull requests less likely to be accepted? The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men! == Do women's pull request acceptance rates start low and increase over time? == [B]etween 1 and 64 pull requests, women's higher acceptance rate remains. Thus, the evidence casts doubt on our hypothesis. == Are women making pull requests that are more needed? == [T]he result suggests that women's increased success rate is not explained by making more immediately needed pull requests. == Are women making smaller changes? == (...) Women make pull requests that add and remove more lines of code, modify more files, and contain more commits. == Are women's pull requests more successful when contributing code? == (For instance, changes to HTML could be more likely to be accepted than changes to C code, and if women are more likely to change HTML) [W]omen's acceptance rates dominate over men's for every programming language in the top ten, to various degrees == Is a woman’s pull request accepted more often because she appears to be a woman? == For insiders, we observe little evidence of bias when we compare women with gender neutral profiles and women with gendered profiles, since both have about equivalent acceptance rates. For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong. Women have a higher acceptance rate of pull requests overall (as we reported earlier), but when they’re outsiders and their gender is identifiable, they have a lower acceptance rate than men. I hope that helps those who won't read the whole paper but feel the need to comment on it.
- luso_brazilian 11y ago
merb 11y ago
I will never get such articles? Why does somebody need to know of which gender is somebody? People should ALWYAYS be measured by their performance not by their gender.
- sp332 11y ago
  When you're contributing to a project, performance is only one important thing. It also matters whether you are furthering the goals of the project, communicating well with the other members, and even if you have the social skills to lead a project.
  - douche 11y ago
    Coming soon: If you possess personal political, social, religious and philosophical views deemed acceptable by the project.
  - merb 11y ago
    that counts to performing good in a job, too. It's just I hate people who have biases against, anything. What I really wanted to say is that I don't get these articles, people should never evaluate people against their outside. no matter if an IT job or anything. the outside appearance has nothing todo with how you really are. and i dislike it so much. I mean yes I know that some groups of people can't go well with others, but thats just something in our minds, because of our past.
- minimaxir 11y ago
  That is not how the real world operates. Unfortunately. There is a reason many startups are teaching awareness of unconscious gender biases.
- anigbrowl 11y ago
  Because it's a well-known factor for discrimination, to the point of limiting people's opportunities to perform in the first place. Until there's rough gender parity that resembles the distribution of gender in the general population fairly closely, you're likely to see significant structural inequalities that hinder open participation in the first place. Myself, I view this as a big problem and think it needs to b researched and actively mitigated. I'm a man, but I score much more like a woman on transgender-related psychological scales despite not taking hormones or engaging in SRS. When I play online videogames I sometimes identify as female (based largely on whim) and I can't help noticing that my female avatars have to put up with an exponentially larger amount of random bullshit from other players. I'm not cool with this, but neither do I want to live in a world where people feel the need to obscure or neutralize their gender, since that's a fairly fundamental part of most people's identity.
wbhart 11y ago
There's a large drop for both men and women between project outsiders who do not identify a gender on GitHub and those that do. So that's one very clear effect. If you want to have your contributions accepted on GitHub, male or female, and you are an outsider, do not identify yourself. However, it was only possible to identify the gender of about a third of people overall from publicly available information. So for the conclusions of the study to be valid, one must eliminate the possibility that there is a gender imbalance in those whose gender is able to be identified by publicly available information (which would not represent discrimination, since that information is self-reported). Until one can eliminate this, one may not be measuring discrimination, but effectively just a correlation between availability of public information on programmers and their skill level. Moreover, it looks like the gender information was entirely constructed from Google+. Yet there is no information on what percentage of users' of each gender actually had Google+ pages. This is extremely relevant to the conclusions as there could also be correlations with the usage of Google+ by programmers of a certain skill and gender.
minimaxir 11y ago
> Specifically, we extract users’ email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users’ profiles. I was curious so I looked up the Google+ API (https://developers.google.com/+/web/api/rest/ https://developers.google.com/+/web/api/rest/) and I do not see an "email lookup" endpoint. You can only search for generic information in public profiles. Searching for my own email, which is public in Google+, using that particular endpoint yielded no results. I call shennanigans that 1/3 of the accounts in the dataset had discoverable profiles this way, and they were retrieved in a manner that is acceptable by Google.
- Xorlev 11y ago
  You can find a G+ profile for anyone by searching an email in your Gmail search. Unsure if that's the route they'd taken.
  - minimaxir 11y ago
    Since that scraping of millions of users would have to be automated, it definitely falls into the issue why Google would allow that behavior.
gregdoesit 11y ago
> Women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable What is also true from the graphs, but the author does not emphasize is this: Men's acceptance rates are 69%, but drop to 64% when the gender is identifiable. This is taken from figure 5. So in general, it seems the acceptance rate drops in both cases, as soon as the gender is identifyable. I do wonder why this is? The two just don't add up. It seems that the type of PRs that the authors have maanged to have the gender identified, simply have much lower acceptance rates.
- civilian 11y ago
  Maybe, when you reveal things about yourself, people are more likely to find things they dislike about you. Having a recognizable gender on github would often include things like: a picture (your face is ugly!), your name (your ethnicity sucks!), your country (your country sucks!), your place of work (your company sucks!) and maybe your side project blog (your side projects are stupid!).
  - gregdoesit 11y ago
    That is a possible explanation. Naively I would have thought the opposite - the more you know about someone, the more likely you accept their PR. But this is a statistically wide enough gap for both cases, and also a statistically more significant drop in acceptance of women's PRs.
    - noddingham 11y ago
      I would say "the more you know about someone like you"... To me that would correlate with other studies based on first impressions, hiring, etc., where we sometimes subconsciously seek out people like us. Similar to another comment, giving any information about you allows me to put you in a box I have created, whether positive or negative.
  - V-2 11y ago
    Or maybe more experienced/open-source involved programmers have a stronger proclivity for choosing abstract user handles? There's no end to this guessing game, all they revealed was some correlation (and not a very strong one at that), and this by itself means next to nothing. Obligatory: http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations
cjensen 11y ago
To me, the real horror show is the 1:18 ratio between contributors who publicly identify themselves as a Woman and contributors who publicly identify themselves as a Man[1]. There's so many hypotheses one could make about that (ratios among software engineers in general, differing interest in programming as a hobby beyond work, the confrontational culture among projects, hiding ones identity to avoid creeps, etc). It's a good start and we could really use further research. [1] Sorry that's worded so awkwardly, but it's hard to phrase that without implying something unsupported by this particular data.
- sheepmullet 11y ago
  > To me, the real horror show is the 1:18 ratio between contributors who publicly identify themselves as a Woman and contributors who publicly identify themselves as a Man[1]. To me, comments like yours highlight the toxic nature of the software industry. It is an expectation in software dev that we do most of our learning and development on our own time and part time work is generally frowned upon. GitHub accounts and side projects are becoming a larger part of interview processes. How do you think this looks to prospective female employees? Basically it gives the impression that they have to be prepared to work 45-50 hours a week in order to be competitive. This isn't compatible with many women's focus on flexible work for when they have children. Notice that none of this is inherent to software development. Companies could provide proper on the job learning and development. Companies could be more open to part time work. Companies could use better proxies for developer skill than personal contributions to GitHub.
  - sheepmullet 11y ago
    If you really want to encourage women then help improve the industry by pushing for 20% time, pushing for flexible work practices, pushing for internal promotions, pushing for salaries to be adjusted to better reflect market rates when they get out of alignment, etc.
  - cjensen 11y ago
    I think you are reading stuff into my comment that I simply did not say or imply; worse, the things you think I said you describe as "toxic"! I don't think side projects are necessary to further your career, and it's been 20 years since I worked on one myself. You do make good points that an individual may not have the opportunity to engage in a side project for reasons not related to competence. Your hypothesis that the hardship faced by many women once they have children may contribute to the statistic is well-worth further investigation -- like I said, there's many possible hypotheses to investigate, and more than one factor may contribute to the overall outcome. As an interviewer, I've only once seen an outside project on a resume and I considered it a plus; that may have been a mistake on my part.
daenz 11y ago
Their method of identifying genders is to use a data source of Github user emails and cross reference those with a Google+ account. Then they scrape the Google+ account and attempt and try to automatically determine the gender. Using this method, they are able to identify only 35% of the users involved in these PRs. I find this suspect, because, anecdotally, I can look almost any author of a PR and determine their gender to a high degree of certainty. Probably well over 90% of the time, just from their name + profile picture + handle. Try it for yourself..look at the latest commits on a random project and see how obvious the genders are most of the time. So the claim that "when a woman is identifiable, PRs are merged less" is totally suspect, because they themselves can only identify the genders of a small percentage relative to what a normal human can identify. If people can identify the genders way more often and accurately, then the claim being made is bogus. Perhaps there is a correlation of strongly signalling your gender (to the point where an inaccurate method of gender-identification has no problem) to being a below average developer.
- astine 11y ago
  Only 35% percent of the accounts have their gender listed in a linked Google+ account. Checking someone's social media profile is a relatively sure way of automatically determining the gender of a lot of people. The authors did use another automated tool to see if they could figure out the gender of users from their Github profile as well, which is something they needed for the second part of their analysis. They don't specify how accurate that procedure was, so it's possible that they are more accurate than you think. Of course, there is still the issue that they have effectively limited their sample to people with Google+ accounts which may affect the results of the study. Given that men's acceptance rate also dropped when their gender was identifiable (but not by as much) gives credence to the idea that there might be a flaw in their Github profile analyzer.
- marcammann 11y ago
  Well, they used two steps. First they identified a sample set of contributors that have self identified, thus validated to some extent, their gender. Further down they then distinguish between contributors where the gender can be inferred from looking at their name & profile picture. Splitting the group of those 35% which were identified via Google+ into two separate groups - identifiable vs. non-identifiable.
jsharf 11y ago
>Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless. If gender isn't identifiable, why is it surprising at all that women have a higher acceptance rate? This is really annoying me, anyone else notice this?
- kuschku 11y ago
  A higher rate than men is unexpected – if women and men were equally qualified, you’d expect the same ratio for men and women.
iLoch 11y ago
I think we should be very careful how we choose to interpret the results here. There are a lot of unknowns and variables presented in the paper (how they're determining gender, for example.) I'm not sure there's enough data here to create an understanding about the relationship between merge rate and gender. Various external issues could be the cause of the difference (I don't believe either gender is more capable) - for example (not sure if this is true, but it portrays how these stats could be skewed): In India, there's a higher incentive to be recognized in the community because competition is stiffer, so more people submit PRs. In this case, people who generally wouldn't submit a PR because of their lack of expertise are now submitting PRs on the chance that it may get accepted. The result is a greater number of PRs with lower quality. These are the effects of having a global community. There are so many external factors I don't really think we can determine much of anything from these results. It's an interesting analysis regardless though.
belorn 11y ago
"For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong." Why leave out an exact number for men whose gender is identifiable?
thezealot 11y ago
It's called a "pussy pass"
jongraehl 11y ago
For regular contributors, women appear* to accept a man's work (slightly) more than they will a woman's, and men appear to accept a woman's work more than they will a man's. Since there are more men than women, this means that on the whole women are privileged. * appear: maybe men or women have a different internal bar for how polished they'll make a pull request (how afraid they are of rejection, etc). the study looks at profiles that are closeted vs out as a gender. If you reveal gender on purpose, this tells us something about you, presumably, but for convenience the difference between closeted and out is taken to signify "bias against [out] men/women". Much of this is not statistically significant, probably (study doesn't give enough info, and suspiciously did an "insider" vs "outsider" analysis rather than a pooled analysis, suggesting they didn't like what they found until they split into two pools). For unsolicited outside contributions, closeted men seem to get rejected more than closeted women (men's bar is lower?). Out men get rejected more than closeted men or women. Out women get rejected more than closeted men or women. The key is: for this category, people appear to be less biased against out men than out women (but somehow people prefer contributions from hidden-gender folks?). Anyway, this is interesting stuff but I'm not sure what to take from it. I do expect more low-quality outside-team submissions from men than women, and I might judge them unfairly if they were out men, but this is just a random hunch and I doubt it would affect me much (probably I wouldn't notice).

177 points by schmichael 11y ago

We considered a GitHub profile as gender neutral if all of the following conditions were met: - an identicon (rather than a profile image) was used, - the gender inference tool output a ’unknown’ for the user’s login name and display name, and - none of the panelists indicated that they could identify the user’s gender. Across both panels, panelists inspected 3000 profiles of roughly equal numbers of women and men. Given that acceptance rates dropped for both men and women when this method determined that a user was not gender-neutral, I wonder if use of a profile picture or real name is the real killer here. Perhaps the kind of person to include identifying information in their Github profile is less likely to have good pull requests? More likely, perhaps having any identifying information at all gives people an excuse to reject your pull requests? That would explain why there was a drop for both sexes but a slightly stronger one for women. If you are identifiable from your profile image or name, people are more likely to reject your PR on the bases of your ethnicity, appearance, religion, or gender, with gender being only a part of the effect.

kuschku 11y ago
> Perhaps the kind of person to include identifying information in their Github profile is less likely to have good pull requests? Probably commercial employees?

zamalek 11y ago

Open Closed Merged Merge Rate 95% Confidence Interval Women 8,216 21,890 111,011 78.6% [78.45%, 78.87%] Men 150,248 591,785 2,181,517 74.6% [74.56%, 74.67%] > The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men! This difference is statistically significant (p < .001). Many hypothesis are discussed in an attempt to explain this and the paper is well worth the read. Thanks to a documented GitHub API this stuff gets quite scientifically correct - everything is backed up with real-world data. More quotes: > To summarize this paper’s observations: > 1. Women are more likely to have pull requests accepted than men. > 2. Women continue to have high acceptance rates as they gain experience. > 3. Women’s pull requests are less likely to serve an immediate project need. > 4. Women’s changes are larger. > 5. Women’s acceptance rates are higher across programming languages. > 6. Women have lower acceptance rates as outsiders when they are identifiable as women.

xbryanx 11y ago
[Edit] Retracting my comment below, after the parent comment was edited to provide more context. --- This quote doesn't seem representative of the whole story that is easily discoverable in the abstract. > Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless.
- jimrandomh 11y ago
  This is indeed in their abstract, but it's misleading. Take a look at page 15 of the preprint. Their data shows that gendered profiles have lower acceptance rates for both genders; that is, pull requests from a gender-neutral profile are more likely to be accepted regardless of the gender of the person behind it. This paper gives the impression that the authors started out hoping to find evidence of discrimination in the form of a lower pull acceptance rate for women than men, they found the opposite, and then they data mined until they could find something they could use to argue there's an anti-woman bias.
tyingq 11y ago
Except, later.... "For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable."
- nightski 11y ago
  Ignoring the fact that this difference could easily be explained away by some other factor when it is considered. It may have nothing to do with it. Causal relationships are so tricky when dealing with statistics.
- jjcm 11y ago
  I'm confused - how does that add up to the average of 78.6% then?

tomjen3 11y ago

Can we please try not to make the comments ending up in a giant cluster-fuck, despite how passionate we feel about the subject?

Analemma_ 11y ago
You must be new here.
- booop 11y ago
  Well, his account is 1937 days older than yours.
  - ashi 11y ago
    It is a common expression suggesting one isn't familiar with the ways of the site/community. You know, for humour.

kstrauser 11y ago

I got to > Our results suggest that although women on GitHub may be more competent overall and stopped. No. Just, no. I'm inclined to disbelieve any study that claims one gender is better at programming than other, and that claim makes me automatically dismiss the rest of its conclusions.

StavrosK 11y ago
You're being downvoted, but what if the result was "Our results suggest that although men on GitHub may be more competent overall"? Then you'd probably be upvoted. Also, I refuse to believe that genders are obviously different in various physiological and mental traits (e.g. height, spatial awareness, ability to host a baby) except the ones that are socially uncomfortable to us.
- mucker 11y ago
  No, that is very unlikely.
- deleted 11y ago
  [deleted]
- kstrauser 11y ago
  I suspect you're right about the first part. That makes me sad, because I'd hope we'd be more even-keeled as a community. I haven't seen any evidence that the second is true, though. I've not heard of a plausible mechanism by which - all else equal - men would be better at abstract reasoning than women or vice versa.
  - StavrosK 11y ago
    I haven't seen any evidence that it's false, though, either, and there are obviously physical and mental differences between sexes/races/whatever, so it seems more likely to me that this would follow the rule, rather than be the exception.
mucker 11y ago

mark-ruwt 11y ago

There's a lot in here, and many of the findings will be different than one would expect from the title. If you don't have much time, skip to the "Discussion" section on pages 16 and 17.

mucker 11y ago

The abstract, despite the clear results, is written in the most hostile manner possible.

jcoffland 11y ago
You must be easily offended. I did not find it hostile.
- pluma 11y ago
  I think the proper idiom is "pouring gasoline on the fire". Especially considering the authors say they intentionally cut the analysis "to keep it crisp" when the analysis actually disagreed with the spirit of the abstract.
  - mucker 11y ago
    Exactly. The title intentionally conflicts with the theme.

Sir_Substance 11y ago

Remember when people on the internet werent pushed towards using real names? Gender bias didn't exist because only people you told knew your gender. Such gay times they were!

pluma 11y ago
> Such gay times they were! "When men were men, women were men and 12 year old girls were FBI agents"
- deleted 11y ago
  [deleted]
jacalata 11y ago
It was so awesome when you were fine if you either were male or knew enough to pretend to be male!
- Sir_Substance 11y ago
  If that was how it was for you, you were hanging out in a pretty crummy part of the internet, which is really sad considering how easy it is to find a different part. You missed out on some quality times.
  - jacalata 11y ago
    That is a completely different argument to the one you made originally - "some parts of the internet weren't sexist" vs "people weren't sexist because they couldn't figure out your gender" (which is the one I disagreed with")
anigbrowl 11y ago
Gender bias didn't exist Yes, I do remember that (having been using the internet several years before there was any such thing as a web browser), and I dispute your claim, albeit on the purely anecdotal basis of spending years on Usenet.

Rangi42 11y ago

> Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles. I wonder if there's a gender bias with regard to making your Google+ profile public (or having one in the first place).

protomyth 11y ago
I get the feeling the peer review is going to flesh some of these issues out. I would love to have time to go over their sample selection.
groby_b 11y ago
I also wonder how they actually verified the public G+ profiles. I know that a significant number of women in my circle of acquaintances deliberately flag themselves as male.
- markus2012 11y ago
  I randomly lie about my gender, age, and anything that could be used to identify me. I learned this from others, and have also encourage my spouse/children/etc. to do the same (which they seem to do). There is no compelling reason for me to provide any real information about myself online.

deleted 11y ago

[deleted]

melted 11y ago

Sigh. Authors are unaware of Simpson's paradox. Such analyses are meaningless if you don't control for such obvious factors as the distributions of projects people of different genders choose to contribute to. See the classic college admissions bias study to understand why: https://en.m.wikipedia.org/wiki/Simpson%27s_paradox https://en.m.wikipedia.org/wiki/Simpson%27s_paradox

dang 11y ago
You have a good point, but you've used the format of a middlebrow dismissal to express it—i.e. acting like others don't know something well-known, and snarking about it. That makes it a bad HN comment even if you're right. A better HN comment would bring up the issue neutrally and then either examine the OP to see how it deals with it, or ask a question about how it does.
- deleted 11y ago
  [deleted]
  - deleted 11y ago
    [deleted]
- V-2 11y ago
  Grandparent, you should have said something along the lines of "authors didn't give an indication of knowing the Simpson's paradox even though their paper was a great opportunity to do so". Or hear from forum lawyers :)
minimaxir 11y ago
Can you give an example of a type of project which would theoetically have different gender distributions of contributors than other projects? That doesn't seem obvious to me.
- melted 11y ago
  Have no idea. But also have an inkling that distributions would differ (and not necessarily the way you'd expect) in projects that require a specialized skill set, such as e.g. top notch design skills or knowledge of DB internals and such. The point being, as stated the whole thing very much resembles the Berkeley gender discrimination study, so proposed improvements are quite obvious: break things down further by project, see if you can glean something from per-project (or at least per-domain) distribution. There might be nothing there, mind you, and the stated conclusions might still hold, but without controlling for the confounding factor there's no way to tell, and the conclusion of this study is therefore not above suspicion.

sergiotapia 11y ago

>Our analysis (not in this paper -- we've cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women.

cushychicken 11y ago

A good start is to call out misogyny when you see it, online or anywhere else. This is disappointingly rare, in my experience. Edit: what a shotgun blast of downvotes and straw men. People are lining up to say all sorts of utter bullshit. For example: * "Where does this indicate misogyny?" * "The paper is about GitHub, which has a reputation for the opposite." * "A good start is to stop calling things misogyny just because you're bad at reading statistics." * "Of course it's important to recognize when it's actual misogyny and not just disagreement with the code." The paper's findings are clear as a fucking bell: women have equal acceptance of pull requests, except when the pull request obviously comes from a woman. Fucking TEXTBOOK misogyny, and you're an idiotic trog if you don't think so. (Looking RIGHT AT YOU here, pluma.)

Zikes 11y ago
However rare it might be, the devastating affects of false positives make this a terrible idea. Just ask Tim Hunt, Matt Taylor, or the men targeted by "donglegate".
- mjb394 11y ago
  Anti-sexist reactions are very rare compared to instances of sexist behavior in my experience. If anti-sexist behavior was as rampant and casual as sexist behavior, those kind of blowups wouldn't happen.
  - Zikes 11y ago
    There are only two appropriate ways to handle it: if the offense is not egregious, address the incident with the offender in private. If the offense is egregious, or if you do not feel comfortable enough to discuss it with the offender, then you should report it to whatever authority figures are responsible for the context of the incident. "Calling out" the behavior consistently creates vigilantism, and puts an unfair burden on the accused. In instances of sexism I have never, ever seen the accused treated as "innocent until proven guilty" in the eyes of the online hate mobs. Tim Hunt's experience was bad enough that he admitted to thoughts of suicide. This is a man fully respected in his field and has a knighthood, but still had difficulty facing the onslaught of hatred.

noddingham 11y ago

The summary seems to be all positive regarding women's contributions except for the last one: "Women have lower acceptance rates as outsiders when they are identifiable as women.". Looking at the data, they say that the acceptance rate for female outsiders is 62.5%, but they don't say what it is for men. Looking at the chart it looks like ~63.5%. And acceptance rates for both genders drop by >=10% going from non-gendered to gendered. Can we infer then that the bias is somewhat across the board for pull requests from easily gendered users?

notatoad 11y ago
it looks like the criteria for determining gender is to pull it from a google+ profile that shares an email address with the github account, and is not using any information from the github profile. It seems likely to me that the bias here is gmail users versus non-gmail (corporate domain?) users.
- Lawtonfogle 11y ago
  Hmm..., this could very well modify the results. Do they present the numbers when they limit it to only people with google+ accounts or to only people using @gmail as their email address?
- astine 11y ago
  That's incorrect. The study identified the gender of the Github users twice. They got their 'canonical' gender for users using your method (and removed from their sample users who could not be identified with this method,) but later to figure out whether a user's gender was identifiable, they used the Github user's own profile. Non-Google+ users were not analyzed at all. That could possibly still influence the results, but not the way you are suggesting.
gregdoesit 11y ago
It does seem like this would be an even more important conclusion. Whether you're a man or a women, your PR will much more likely be accepted of your profile doesn't reveal your gender (at least not in the sense that creators of this study can identify it).

4bpp 11y ago

Given that it seems like they analysed real-world data rather than producing synthetic experiments, I wouldn't put too much stock in the findings. For instance, in a hypothetical world where all businesses contributing to Github projects are perfectly sexist, only ever assign male coders to core engineering and only ever assign female coders to UX design, UX design is intrinsically more controversial and Github maintainerers accept patches purely on the degree of public support for their contents, one would expect a far lower degree of acceptance for patches from identifiable (as businesses presumably make their workers operate under real-name accounts) female accounts without this containing any signal about sexism on the Github side. A better methodology for a study like this would be to identify a number of people planning to make sporadic contributions to projects they have no connection to and making them submit the pull requests under fresh fictional profiles with randomised gender. This way, any irrelevant correlations between the kind of pull request being made and the requesting party being identifiable as female would be eliminated.

nsfyn55 11y ago
I would tend to agree with you. When you use public data you get all the implied imperfections of public data. You can make some vague inferences but since the controls aren't in laboratory conditions the potential for biased results is too high. If they showed individuals with already high acceptance rates received lower acceptance rates on the same code when submitted from a gender identifiable account then I'd find it to be more compelling.
johnhess 11y ago
As the article itself states: > Experiments and retrospective field studies each have advantages. The advantage of experiments is that they can more confidently infer cause and effect by isolating gender as the predictor variable. The advantage of retrospective field studies is that they tend to have higher ecological validity because they are conducted in real-world situations. In this paper, we use a retrospective field study as a first step to quantify the effect of gender bias in open source. The research is presented in context, an important and valid context, too. At the end, they discuss potential explanations and never go so far as to say "And all of this is clearly because ______". A controlled study would be a great followup, but I'd sure put stock in these compelling and well documented findings.

BinaryIdiot 11y ago

Remember this is not peered reviewed. Others haven't looked at it to decide if it had proper controls, good enough samples, or even valid data. So take this with a grain of salt until it's properly peered reviewed. I almost wonder if it's worse to publish non-peered-reviewed research or not. I mean it could be great if it's solid research but if there are flaws (even minor) it could undermine other studies, change people's minds, etc.

luso_brazilian 11y ago

Here is one (IMO) much more complete abstract (not to call it tl:dr) that covers the main hypothesis, and all the variables studied Our main research question was: to what extent does gender bias exist among people who judge GitHub pull requests? To answer this question, we approached the problem by examining whether men and women are equally likely to have their pull requests accepted on GitHub, then investigated why differences might exist. We extract users' email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users' profiles. Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles. == Results == Are women's pull requests less likely to be accepted? The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men! == Do women's pull request acceptance rates start low and increase over time? == [B]etween 1 and 64 pull requests, women's higher acceptance rate remains. Thus, the evidence casts doubt on our hypothesis. == Are women making pull requests that are more needed? == [T]he result suggests that women's increased success rate is not explained by making more immediately needed pull requests. == Are women making smaller changes? == (...) Women make pull requests that add and remove more lines of code, modify more files, and contain more commits. == Are women's pull requests more successful when contributing code? == (For instance, changes to HTML could be more likely to be accepted than changes to C code, and if women are more likely to change HTML) [W]omen's acceptance rates dominate over men's for every programming language in the top ten, to various degrees == Is a woman’s pull request accepted more often because she appears to be a woman? == For insiders, we observe little evidence of bias when we compare women with gender neutral profiles and women with gendered profiles, since both have about equivalent acceptance rates. For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong. Women have a higher acceptance rate of pull requests overall (as we reported earlier), but when they’re outsiders and their gender is identifiable, they have a lower acceptance rate than men. I hope that helps those who won't read the whole paper but feel the need to comment on it.

luso_brazilian 11y ago

merb 11y ago

I will never get such articles? Why does somebody need to know of which gender is somebody? People should ALWYAYS be measured by their performance not by their gender.

sp332 11y ago
When you're contributing to a project, performance is only one important thing. It also matters whether you are furthering the goals of the project, communicating well with the other members, and even if you have the social skills to lead a project.
- douche 11y ago
  Coming soon: If you possess personal political, social, religious and philosophical views deemed acceptable by the project.
- merb 11y ago
  that counts to performing good in a job, too. It's just I hate people who have biases against, anything. What I really wanted to say is that I don't get these articles, people should never evaluate people against their outside. no matter if an IT job or anything. the outside appearance has nothing todo with how you really are. and i dislike it so much. I mean yes I know that some groups of people can't go well with others, but thats just something in our minds, because of our past.
minimaxir 11y ago
That is not how the real world operates. Unfortunately. There is a reason many startups are teaching awareness of unconscious gender biases.
anigbrowl 11y ago
Because it's a well-known factor for discrimination, to the point of limiting people's opportunities to perform in the first place. Until there's rough gender parity that resembles the distribution of gender in the general population fairly closely, you're likely to see significant structural inequalities that hinder open participation in the first place. Myself, I view this as a big problem and think it needs to b researched and actively mitigated. I'm a man, but I score much more like a woman on transgender-related psychological scales despite not taking hormones or engaging in SRS. When I play online videogames I sometimes identify as female (based largely on whim) and I can't help noticing that my female avatars have to put up with an exponentially larger amount of random bullshit from other players. I'm not cool with this, but neither do I want to live in a world where people feel the need to obscure or neutralize their gender, since that's a fairly fundamental part of most people's identity.

wbhart 11y ago

There's a large drop for both men and women between project outsiders who do not identify a gender on GitHub and those that do. So that's one very clear effect. If you want to have your contributions accepted on GitHub, male or female, and you are an outsider, do not identify yourself. However, it was only possible to identify the gender of about a third of people overall from publicly available information. So for the conclusions of the study to be valid, one must eliminate the possibility that there is a gender imbalance in those whose gender is able to be identified by publicly available information (which would not represent discrimination, since that information is self-reported). Until one can eliminate this, one may not be measuring discrimination, but effectively just a correlation between availability of public information on programmers and their skill level. Moreover, it looks like the gender information was entirely constructed from Google+. Yet there is no information on what percentage of users' of each gender actually had Google+ pages. This is extremely relevant to the conclusions as there could also be correlations with the usage of Google+ by programmers of a certain skill and gender.

minimaxir 11y ago

> Specifically, we extract users’ email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users’ profiles. I was curious so I looked up the Google+ API (https://developers.google.com/+/web/api/rest/ https://developers.google.com/+/web/api/rest/) and I do not see an "email lookup" endpoint. You can only search for generic information in public profiles. Searching for my own email, which is public in Google+, using that particular endpoint yielded no results. I call shennanigans that 1/3 of the accounts in the dataset had discoverable profiles this way, and they were retrieved in a manner that is acceptable by Google.

Xorlev 11y ago
You can find a G+ profile for anyone by searching an email in your Gmail search. Unsure if that's the route they'd taken.
- minimaxir 11y ago
  Since that scraping of millions of users would have to be automated, it definitely falls into the issue why Google would allow that behavior.

gregdoesit 11y ago

> Women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable What is also true from the graphs, but the author does not emphasize is this: Men's acceptance rates are 69%, but drop to 64% when the gender is identifiable. This is taken from figure 5. So in general, it seems the acceptance rate drops in both cases, as soon as the gender is identifyable. I do wonder why this is? The two just don't add up. It seems that the type of PRs that the authors have maanged to have the gender identified, simply have much lower acceptance rates.

civilian 11y ago
Maybe, when you reveal things about yourself, people are more likely to find things they dislike about you. Having a recognizable gender on github would often include things like: a picture (your face is ugly!), your name (your ethnicity sucks!), your country (your country sucks!), your place of work (your company sucks!) and maybe your side project blog (your side projects are stupid!).
- gregdoesit 11y ago
  That is a possible explanation. Naively I would have thought the opposite - the more you know about someone, the more likely you accept their PR. But this is a statistically wide enough gap for both cases, and also a statistically more significant drop in acceptance of women's PRs.
  - noddingham 11y ago
    I would say "the more you know about someone like you"... To me that would correlate with other studies based on first impressions, hiring, etc., where we sometimes subconsciously seek out people like us. Similar to another comment, giving any information about you allows me to put you in a box I have created, whether positive or negative.
- V-2 11y ago
  Or maybe more experienced/open-source involved programmers have a stronger proclivity for choosing abstract user handles? There's no end to this guessing game, all they revealed was some correlation (and not a very strong one at that), and this by itself means next to nothing. Obligatory: http://www.tylervigen.com/spurious-correlations http://www.tylervigen.com/spurious-correlations

cjensen 11y ago

To me, the real horror show is the 1:18 ratio between contributors who publicly identify themselves as a Woman and contributors who publicly identify themselves as a Man[1]. There's so many hypotheses one could make about that (ratios among software engineers in general, differing interest in programming as a hobby beyond work, the confrontational culture among projects, hiding ones identity to avoid creeps, etc). It's a good start and we could really use further research. [1] Sorry that's worded so awkwardly, but it's hard to phrase that without implying something unsupported by this particular data.

sheepmullet 11y ago
> To me, the real horror show is the 1:18 ratio between contributors who publicly identify themselves as a Woman and contributors who publicly identify themselves as a Man[1]. To me, comments like yours highlight the toxic nature of the software industry. It is an expectation in software dev that we do most of our learning and development on our own time and part time work is generally frowned upon. GitHub accounts and side projects are becoming a larger part of interview processes. How do you think this looks to prospective female employees? Basically it gives the impression that they have to be prepared to work 45-50 hours a week in order to be competitive. This isn't compatible with many women's focus on flexible work for when they have children. Notice that none of this is inherent to software development. Companies could provide proper on the job learning and development. Companies could be more open to part time work. Companies could use better proxies for developer skill than personal contributions to GitHub.
- sheepmullet 11y ago
  If you really want to encourage women then help improve the industry by pushing for 20% time, pushing for flexible work practices, pushing for internal promotions, pushing for salaries to be adjusted to better reflect market rates when they get out of alignment, etc.
- cjensen 11y ago
  I think you are reading stuff into my comment that I simply did not say or imply; worse, the things you think I said you describe as "toxic"! I don't think side projects are necessary to further your career, and it's been 20 years since I worked on one myself. You do make good points that an individual may not have the opportunity to engage in a side project for reasons not related to competence. Your hypothesis that the hardship faced by many women once they have children may contribute to the statistic is well-worth further investigation -- like I said, there's many possible hypotheses to investigate, and more than one factor may contribute to the overall outcome. As an interviewer, I've only once seen an outside project on a resume and I considered it a plus; that may have been a mistake on my part.

daenz 11y ago

Their method of identifying genders is to use a data source of Github user emails and cross reference those with a Google+ account. Then they scrape the Google+ account and attempt and try to automatically determine the gender. Using this method, they are able to identify only 35% of the users involved in these PRs. I find this suspect, because, anecdotally, I can look almost any author of a PR and determine their gender to a high degree of certainty. Probably well over 90% of the time, just from their name + profile picture + handle. Try it for yourself..look at the latest commits on a random project and see how obvious the genders are most of the time. So the claim that "when a woman is identifiable, PRs are merged less" is totally suspect, because they themselves can only identify the genders of a small percentage relative to what a normal human can identify. If people can identify the genders way more often and accurately, then the claim being made is bogus. Perhaps there is a correlation of strongly signalling your gender (to the point where an inaccurate method of gender-identification has no problem) to being a below average developer.

astine 11y ago
Only 35% percent of the accounts have their gender listed in a linked Google+ account. Checking someone's social media profile is a relatively sure way of automatically determining the gender of a lot of people. The authors did use another automated tool to see if they could figure out the gender of users from their Github profile as well, which is something they needed for the second part of their analysis. They don't specify how accurate that procedure was, so it's possible that they are more accurate than you think. Of course, there is still the issue that they have effectively limited their sample to people with Google+ accounts which may affect the results of the study. Given that men's acceptance rate also dropped when their gender was identifiable (but not by as much) gives credence to the idea that there might be a flaw in their Github profile analyzer.
marcammann 11y ago
Well, they used two steps. First they identified a sample set of contributors that have self identified, thus validated to some extent, their gender. Further down they then distinguish between contributors where the gender can be inferred from looking at their name & profile picture. Splitting the group of those 35% which were identified via Google+ into two separate groups - identifiable vs. non-identifiable.

jsharf 11y ago

>Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless. If gender isn't identifiable, why is it surprising at all that women have a higher acceptance rate? This is really annoying me, anyone else notice this?

kuschku 11y ago
A higher rate than men is unexpected – if women and men were equally qualified, you’d expect the same ratio for men and women.

iLoch 11y ago

I think we should be very careful how we choose to interpret the results here. There are a lot of unknowns and variables presented in the paper (how they're determining gender, for example.) I'm not sure there's enough data here to create an understanding about the relationship between merge rate and gender. Various external issues could be the cause of the difference (I don't believe either gender is more capable) - for example (not sure if this is true, but it portrays how these stats could be skewed): In India, there's a higher incentive to be recognized in the community because competition is stiffer, so more people submit PRs. In this case, people who generally wouldn't submit a PR because of their lack of expertise are now submitting PRs on the chance that it may get accepted. The result is a greater number of PRs with lower quality. These are the effects of having a global community. There are so many external factors I don't really think we can determine much of anything from these results. It's an interesting analysis regardless though.

belorn 11y ago

"For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong." Why leave out an exact number for men whose gender is identifiable?

thezealot 11y ago

It's called a "pussy pass"

jongraehl 11y ago

For regular contributors, women appear* to accept a man's work (slightly) more than they will a woman's, and men appear to accept a woman's work more than they will a man's. Since there are more men than women, this means that on the whole women are privileged. * appear: maybe men or women have a different internal bar for how polished they'll make a pull request (how afraid they are of rejection, etc). the study looks at profiles that are closeted vs out as a gender. If you reveal gender on purpose, this tells us something about you, presumably, but for convenience the difference between closeted and out is taken to signify "bias against [out] men/women". Much of this is not statistically significant, probably (study doesn't give enough info, and suspiciously did an "insider" vs "outsider" analysis rather than a pooled analysis, suggesting they didn't like what they found until they split into two pools). For unsolicited outside contributions, closeted men seem to get rejected more than closeted women (men's bar is lower?). Out men get rejected more than closeted men or women. Out women get rejected more than closeted men or women. The key is: for this category, people appear to be less biased against out men than out women (but somehow people prefer contributions from hidden-gender folks?). Anyway, this is interesting stuff but I'm not sure what to take from it. I do expect more low-quality outside-team submissions from men than women, and I might judge them unfairly if they were out men, but this is just a random hunch and I doubt it would affect me much (probably I wouldn't notice).