5 ms·

Blind Reviews Experiment to remove code reviewer bias on Bugzilla and GitHub

24 points by cpeterso 9y ago

cheez 9y ago
Great idea in general. Software dev has an unnecessary amount of politics, and this could reduce it.
- grv07 9y ago
  Yes, this improves the environment :).
codingdave 9y ago
Interesting idea, but everyone I've worked with has their own distinct flavors of how they code, so this wouldn't really hide identity much, not unless you didn't know the coder in the first place.
TheChaplain 9y ago
This seems like a problem that needs a cultural solution, not a technical one? Because knowing who submitted the code, I can better manage my time. E.g. if the submitter is known for producing high quality code, I only need to spend minimal time browsing through it. But, if the submitter is known to cut corners I can prepare myself to spend more time and review each line carefully because I most certainly will find issues.
- jadedhacker 9y ago
  I think it could be helpful for screening first time contributions at the very least. You're right that a cultural solution is necessary, but in the mean time one way to deal with bias when you know you are biased is to remove the ability for it to operate (when practicable). Otherwise, you're stuck in this zone where you know you're liable to make bad decisions, but you rationalise to yourself that you can correct them. No matter how you slice it, it's never going to be possible to exactly correct for your biases, you're going to undershoot or overshoot. Better to minimize their scope of operation. I really like this line of thinking because it crashes into this realm of philosophy where some people think you can feel your way to a good result, whereas the other approach is to initially intuit the just result, and rationally design a system that achieves it regardless of your instantaneous emotions. Of course, such systems thinking is very very sensitive to getting the right intuition up front, so it's very important to train it and build in feedback systems in case you start getting things you didn't expect.
- jfoutz 9y ago
  There is room for both. Code review of high quality code is enlightening. Some percentage should be blind. It’s a classic exploration vs exploitation trade off of your time.
- zilchers 9y ago
  But isn’t that sort of the issue? If you make assumptions about overall code quality, you’ll probably let smaller stylistic things slide because “this is a good coder.” I think about this in terms of some of the more contentious project (think bitcoin, this would be great for removing some of the personality clashes there).
Arete31415 9y ago
FYI, when orchestras started doing blind auditions, the numbers of women hired skyrocketed. Even though you'd think music would be the one area where what a person looked like & whether they were male or female would be irrelevant. I think this is a great idea.
- legostormtroopr 9y ago
  Except sometimes blind hiring doesn't go the direction you are hoping. http://www.abc.net.au/news/2017-06-30/bilnd-recruitment-trial-to-improve-gender-equality-failing-study/8664888 http://www.abc.net.au/news/2017-06-30/bilnd-recruitment-tria...
  - mpweiher 9y ago
    Yep. Also, the orchestra thing was 1952.
  - nerdponx 9y ago
    Article didn't stop to get into why that happened. Were there just more qualified men in the applicant pool than women?
  - tdb7893 9y ago
    Yeah, doing blind hiring won't fix most biases. People's careers are reflective of the oppourtunites that they have had and blind hiring seems like it has the potential to just reinforce existing societal biases instead of eliminating them.
- ubernostrum 9y ago
  This article is relevant for the discussion: https://peerj.com/articles/cs-111/ https://peerj.com/articles/cs-111/ They explored a lot of factors to do with gender in a data set of several years' worth of public pull requests on GitHub. One of the more interesting findings was that women's pull requests get accepted more often, but that a couple factors can strongly affect that, including whether the person reviewing the PR knows the author of the PR is a woman: For outsiders, we see evidence for gender bias: women’s acceptance rates drop by 12.0% when their gender is identifiable, compared to when it is not (χ2(df = 1, n = 16, 258) = 158, p < .001). There is a smaller 3.8% drop for men (χ2(df = 1, n = 608,764) = 39, p < .001). Women have a higher acceptance rate of pull requests overall (as we reported earlier), but when they are outsiders and their gender is identifiable, they have a lower acceptance rate than men.

24 points by cpeterso 9y ago

cheez 9y ago

Great idea in general. Software dev has an unnecessary amount of politics, and this could reduce it.

grv07 9y ago
Yes, this improves the environment :).

codingdave 9y ago

Interesting idea, but everyone I've worked with has their own distinct flavors of how they code, so this wouldn't really hide identity much, not unless you didn't know the coder in the first place.

TheChaplain 9y ago

This seems like a problem that needs a cultural solution, not a technical one? Because knowing who submitted the code, I can better manage my time. E.g. if the submitter is known for producing high quality code, I only need to spend minimal time browsing through it. But, if the submitter is known to cut corners I can prepare myself to spend more time and review each line carefully because I most certainly will find issues.

jadedhacker 9y ago
I think it could be helpful for screening first time contributions at the very least. You're right that a cultural solution is necessary, but in the mean time one way to deal with bias when you know you are biased is to remove the ability for it to operate (when practicable). Otherwise, you're stuck in this zone where you know you're liable to make bad decisions, but you rationalise to yourself that you can correct them. No matter how you slice it, it's never going to be possible to exactly correct for your biases, you're going to undershoot or overshoot. Better to minimize their scope of operation. I really like this line of thinking because it crashes into this realm of philosophy where some people think you can feel your way to a good result, whereas the other approach is to initially intuit the just result, and rationally design a system that achieves it regardless of your instantaneous emotions. Of course, such systems thinking is very very sensitive to getting the right intuition up front, so it's very important to train it and build in feedback systems in case you start getting things you didn't expect.
jfoutz 9y ago
There is room for both. Code review of high quality code is enlightening. Some percentage should be blind. It’s a classic exploration vs exploitation trade off of your time.
zilchers 9y ago
But isn’t that sort of the issue? If you make assumptions about overall code quality, you’ll probably let smaller stylistic things slide because “this is a good coder.” I think about this in terms of some of the more contentious project (think bitcoin, this would be great for removing some of the personality clashes there).

Arete31415 9y ago

FYI, when orchestras started doing blind auditions, the numbers of women hired skyrocketed. Even though you'd think music would be the one area where what a person looked like & whether they were male or female would be irrelevant. I think this is a great idea.

legostormtroopr 9y ago
Except sometimes blind hiring doesn't go the direction you are hoping. http://www.abc.net.au/news/2017-06-30/bilnd-recruitment-trial-to-improve-gender-equality-failing-study/8664888 http://www.abc.net.au/news/2017-06-30/bilnd-recruitment-tria...
- mpweiher 9y ago
  Yep. Also, the orchestra thing was 1952.
- nerdponx 9y ago
  Article didn't stop to get into why that happened. Were there just more qualified men in the applicant pool than women?
- tdb7893 9y ago
  Yeah, doing blind hiring won't fix most biases. People's careers are reflective of the oppourtunites that they have had and blind hiring seems like it has the potential to just reinforce existing societal biases instead of eliminating them.
ubernostrum 9y ago
This article is relevant for the discussion: https://peerj.com/articles/cs-111/ https://peerj.com/articles/cs-111/ They explored a lot of factors to do with gender in a data set of several years' worth of public pull requests on GitHub. One of the more interesting findings was that women's pull requests get accepted more often, but that a couple factors can strongly affect that, including whether the person reviewing the PR knows the author of the PR is a woman: For outsiders, we see evidence for gender bias: women’s acceptance rates drop by 12.0% when their gender is identifiable, compared to when it is not (χ2(df = 1, n = 16, 258) = 158, p < .001). There is a smaller 3.8% drop for men (χ2(df = 1, n = 608,764) = 39, p < .001). Women have a higher acceptance rate of pull requests overall (as we reported earlier), but when they are outsiders and their gender is identifiable, they have a lower acceptance rate than men.