6 ms·
While this might indeed help employers find better employees, I am concerned about hidden (or not so hidden) bias. Games like this (and the way this one is desc
by pron 12y ago
While this might indeed help employers find better employees, I am concerned about hidden (or not so hidden) bias. Games like this (and the way this one is described confirms it) select for competitive people. Other talented programmers might prefer expressing their creativity and talent in less competitive ways. While competitiveness might be a desirable quality, sometimes it is not. It might also appeal to men more than women, and so might be inadvertently sexist. I think you should very carefully analyze player demographics to make sure you're not making matters worse in terms of diversity.
When you design a test -- any test -- you should make sure that 1/ score correlates with success (i.e. full bivariate correlation), and 2/ if the test is voluntary, that there are no prior biases which would select candidates prior to taking the test.
- tptacek 12y agoBest answer I can give was that we've heard that concern from a bunch of different sources, we share it, and we're designing against it to some extent. As a reminder: this is an outreach strategy we deployed at Matasano to enormous success. We had an English professor finish the crypto challenges. We didn't solve all our diversity problems, but we made a palpable dent in them, and we did that by coming up with something that surfaced aptitude that wasn't held hostage to the biases of random human interviewers. There is, to my mind, no hope for the "random human interviewer" hiring strategy. It's hard to make the problem we have now worse. But we're watching out for it. I would love to hear more thoughts on how we can address this at the level of game design! We're at a "finishing touches on infrastructure, just starting with level design" place in our design right now.
- pron 12y agoThe thing is that the effect of such biases is hard to measure directly. Employers will get better employees than before, but may also miss more good employees than before, yet notice the former much sooner than the latter (which will have a delayed effect). The best way to address it is to first measure for biases (run a demographic analysis on players), and second actively design for unselected populations with focus groups. I am sure some of this information has already been collected by game studios.
- yummyfajitas 12y agoSomewhat relatedly, as a person who's discussed these issues with you in the past, it sounds like you'll be gathering new evidence. How will your beliefs change? For example, suppose $GROUP is over/underrepresented among top starfighter performers. What will be your new belief about $GROUP's technical talent and the causes of over/underrepresentation in tech in general?
- pron 12y agoI don't think my beliefs about the facts are any different from yours; we disagree on the causes of the facts and their mutability. Therefore, my beliefs will change when the dynamics are tested. But this kind of data provides evidence for our disagreement no more than a still image of a flying arrow can settle the one between Aristotle's and Newton's theories of motion. Somewhat relatedly I'd like to add that while the result of an experiment in dynamics will obviously change my theory, it will in no way change my values[1]. Privilege based on an immutable characteristic such as intelligence (assuming such a result were to be obtained) is no more arbitrary than one based on bloodline. A smart person, though she will obtain it, deserves no more power than the average bloke, just as a nobleman, though he will obtain it, deserves no more power than a commoner. This is one reason the original meaning of "meritocracy" is satirical[2], as it does not change anything other than for the worse, by making the wielders of power believe that they actually deserve it (like the nobility in ancient times but unlike more recent ruling classes such as the American WASPs). So far I have found that people who believe that the variance of some traits such as intelligence between population groups is dominantly the result of genetics do so because they think it provides a moral justification to the social order: things are as they ought to be because nature dictates so. I see no connection between the two. Nature (if it is, in fact, at play) has little bearing on ethics[3], and thus can, at most, explain but never justify an unfair distribution of power. [1]: http://en.wikipedia.org/wiki/Fact%E2%80%93value_distinction http://en.wikipedia.org/wiki/Fact%E2%80%93value_distinction [2]: http://www.theguardian.com/politics/2001/jun/29/comment http://www.theguardian.com/politics/2001/jun/29/comment [3]: http://en.wikipedia.org/wiki/Naturalistic_fallacy http://en.wikipedia.org/wiki/Naturalistic_fallacy
- yummyfajitas 12y ago
- sireat 12y agoIt is a bias, but it is similar to the following bias: http://www.theonion.com/video/in-the-know-are-tests-biased-against-students-who,17966/ http://www.theonion.com/video/in-the-know-are-tests-biased-a... How do you test for noncompetitive but still productive people using a competitive test?