13 ms·

Overwatch Randomized Trial on Gender

202 points by Aissen 8y ago

jasonshen 8y ago
This is about as well done of a study as we could ask for from a non-academic research level. Basically while the number of sexual comments were relatively low, the amount of questioning/challenging of his abilities went up a lot when he played with a feminine name. Reminds me of a professor at Stanford who underwent a sex change and experienced very different interactions from his peers: https://www.nature.com/scitable/content/does-gender-matter-by-ben-a-barres-10602856 https://www.nature.com/scitable/content/does-gender-matter-b... We recently did a survey of 240 collegiate esports players and found that women were 9x more likely to be "very concerned" about sexism and 2x as likely to be "very concerned" about bullying in competitive gaming culture. I think Dan's research can help explain why. https://www.midgame.gg/survey https://www.midgame.gg/survey
- ToFundorNot 8y ago
  This study is missing one important factor, a baseline. The author should have provided an androgynous named account to see how that fared.
  - Dylan16807 8y ago
    An A/B test doesn't need to attempt a "middle" option. And there's no reason to assume an androgynous name would be in the middle or in any way qualify as a "baseline".
    - zaarn 8y ago
      It would still be very interesting to see how people would react to neutral usernames like randomly generated number sequences. It could provide some useful insight.
      - seqastian 8y ago
        Gamers just default to male.
        diegoperini 8y ago
        Depends on the game but I see your point.
        zaarn 8y ago
        That's just an assertion without data backing it up. The above experiment with neutral names would give some insight if "gamers" do indeed "default to male", as you express it, or not.
- towaway1138 8y ago
  > about as well done of a study as we could ask for from a non-academic research level Unfortunately, there's a good reason why academics use proper experimental design. Without it, all you really have is suggestive noise, if even that.
  - macleginn 8y ago
    The definition of “proper experimental design” varies widely by the field, and DL’s methodology looks quite solid, as far as social sciences and the task at hand are concerned.
    - leereeves 8y ago
      The generally poor quality of research in the social sciences (proven so by the lack of reproducibility) is not really a good defense for the poor quality of any individual study.
      - naasking 8y ago
        The author is pretty rigourous about explaining the limitations.
      - esailija 8y ago
        Yes, it is? Science does not guarantee knowledge in any field, it's only supposed to increase the odds. Are the odds of getting true knowledge in this field increased compared to the alternatives (I.E. people just arbitrarily claiming stuff based on personal anecdotes?) Then it's the best we have in that field and rationally you have no reason to believe the claims of knowledge acquired from worse methods over the better alternatives - just like it works in other fields.
        leereeves 8y ago
        Why do you assume research in the social sciences even increases the odds of a true result? Combining low standards with a publish or perish culture actually encourages the publication of spurious results (and worse behavior) and hinders the careers of researchers with higher standards.
Nexxxeh 8y ago
Really interesting read. I appreciate that it highlights the problems with the data and doesn't reach ridiculous conclusions based on junk data. Anecdotally I've found a noticeable improvement since Endorsements, which this data missed. Also the rise of Fortnite and more recently Apex Legends has led to a further decrease in toxicity that I've seen. A lot of toxic kids left for BR pastures. That's on PC in the EU region on a high-silver current, low-gold peak SR account, casual QP-heavy play but some comp, pure flexing (as I prefer to fill rather than focus on maining one or two characters. Maining is arguably the best way to climb out to Plat). I'm an almost ridiculously positive player with a gender neutral name and I've found being overtly positive vastly improves the experience. It does prompt a couple of toxic responses ("kys", "fuck off" etc) but the majority is really positive. (I generally just report any toxicity.) Playing without voice in Comp is sub-optimal, as winning is much easier with some kind of synergy. Even a basic PharMercy combo, or ultimate combos like a EMP+DVa Bomb can be highly effective. I normally Solo Q, but playing paired with a friend in QP yesterday I had a surreal experience. One player on the enemy team insulted my friend (sexual but not specifically-gendered insult). She was playing with a non-gendered name, as Moira, and the toxic enemy player complained she was carrying us. All the players on both teams turned on the toxic player, calling them out and highlighting why they were actually losing. I have not seen anything like it before but it was hilarious. Stay As A Team is great in theory, but a team of 6 randomised will get rekt by a pre-made 6 stack on Discord with each other, and the matchmaker doesn't seem to take that into account. I wonder what region author was playing on?
- kfwhp 8y ago
  Every time I see a feminine name in a videogame I think the person is just fishing for attention. This is important because most players of both genders use neutral-looking names. The study doesn't account for this.
  - SolaceQuantum 8y ago
    Honestly, I don't really see such behavior directed towards male names though. Stuff like LucasJones or MikeAtLarge or LordOfKilling doesn't get the same attention that SuzieHealsU or CuteLamb or PrincessHurtful might. If it was equal, I'd expect 'MrKickass' to be questioned and assumed to be grabbing attntion that 'MsKilljoy' gets... but in my experience this distinctly isn't the case.
ggggtez 8y ago
Obviously, a number of issues present in the study, but I can appreciate they tried to account for some of them. Lots of variables are unaccounted for like time of day, win/loss, character, what the researcher said in chat, etc. Similarly, the opening preamble points out that often the worst harassment is potentially related to when players hear female voices. Obviously there is a different way you could do this study, which would be to invite ~300 players of various skill levels to play a single game and record the results for analysis. That type of study would have it's own problems, but you could at least be more likely to control for some of the problems in this study.
- mactrey 8y ago
  Yeah, the voice is the key. The gender ratio is skewed so much at higher ranks that an obviously feminine name will simply be assumed to be ironic. I would imagine the toxicity/sexual comments would increase by an order of magnitude if the voice was female-presenting along with the account name.
- novia 8y ago
  Or this researcher could have got a woman to speak into his mic while he played.
AcerbicZero 8y ago
This is an interesting study, and helps highlight why Overwatch needs a scoreboard. With 6 random people on a team, very limited personal statistics, and no game length team statistics, everyone tends to get toxic once you start losing. I've played everything from high gold (tanked rank for a few seasons) to low masters, and its the same pattern every time. Someone makes an accusation about a player not pulling their weight, demands a swap in a rude way, and the player being accused defends themselves. Then the name calling starts. It would be interesting to see what roles the author was playing, and the exact SR of the accounts, as there seems to be a great deal of variation in what people will say on voice comms (usually higher SR, because voice comms > chat) and what will be said in chat. People also like to blame DPS first, then demand a Reinhardt (if they don't have one) and then finally blame support. Without hard data as to what is going wrong with a team composition, toxicity is almost guaranteed when losing.
- quxbar 8y ago
  To be fair to the paper, toxicity is almost guaranteed when losing while playing with a feminine name. It's pretty rare otherwise.
- wlesieutre 8y ago
  > People also like to blame DPS first, then demand a Reinhardt (if they don't have one) and then finally blame support. As someone who mains support, I don't think this is true. I've had plenty of games where I do a shitton of healing, get the healing card at the end, and still had people blaming me for "0 heals" all game. A scoreboard would help with that, but it'd be a trade-off with having 5 angry people shitting on you if you ever have a bad game. I'm not sure that's better. EDIT: Plus there's all the times when your stats are bad but you're not the right one to blame, like when the whole team can't move up through a choke because the tanks are too chicken to go first, but it gets reflected in poor DPS eliminations. Ultimately it's a team game, and failures of teamwork aren't reflected well in a stats scoreboard. I think you'd incentivize poor team play by people trying to pad their own numbers at the team's expense. And no matter what information is available, some people are just looking for someone to blame when they lose. I'm not sure how to fix that. If matchmaking is working right you're going to lose about half the time, there's no point in getting worked up about it but people still do.
NullPrefix 8y ago
>There are a lot of games that weren't included in the experiment because I wasn't in a mood to listen to someone rage at their team for fifteen minutes and the procedure I used involved pre-committing to not muting people who do that.
SolaceQuantum 8y ago
"If that result generalizes to OW, that would explain something which I thought was odd, which was that a lot of demands to switch and general vitriol came during my best performances with the feminine account. " This isn't statistically analyzed, but I think it's a fairly significant detail that's not emphasized more in the paper.
- qiqing 8y ago
  Agreed. Especially this excerpt: > Per above, I didn't want to code tone of messages to avoid bias, so this table only indicates the rate at which people told me I was playing incorrectly or asked that I switch to a different character. The qualitative difference in experience is understated by this table. For example, the one time someone asked me to switch characters in the masculine condition, the request was a one sentence, polite, request ("hey, we're dying too quickly, could we switch [from the standard one primary healer / one off healer setup] to double primary healer or switch our tank to [a tank that can block more damage]?"). When using the feminine name, a typical case would involve 1-4 people calling me human garbage for most of the game and consoling themselves with the idea that the entire reason our team is losing is that I won't change characters.
nickpsecurity 8y ago
Great study! Only gripe is that most trolling I get in online gaming is through voice chat. They also seem to troll me a tad bit more if I have a mic plugged in but I'm not talking. I suggest a re-run of this experiment recording the audio in case we get wildly different results. In a one-party consent state if you want to play it safe. ;) I also think it will vary game by game. I don't play a lot of games. Most are shooters like Modern Warfare, Battlefield, and Mass Effect 3. Modern Warfare 2 and 3 on Xbox brought in more trolls in voice chat than all my others combined except maybe GTA V. In the last one, you play as a vicious thug causing problems for everyone around you. Them being pricks makes more sense. The difference in shooter games, though, was more interesting given it might reflect different sub-cultures in gaming. Specifically, MW is designed to appeal to largest number of people with lowest barrier to entry and patience required. Might self-select for higher percentage of jerks. Whereas, Battlefield 4 and Mass Effect 3 push people to work together more for best results.
- towaway1138 8y ago
  I don't understand why people ever play with voice chat enabled. I don't want to hear streams of ungrammatical foul language, and I certainly don't want to hear people slapping their kids, etc. TMI.
  - jdietrich 8y ago
    It's absolutely necessary if you want to perform well in a team-based game.
  - dragontamer 8y ago
    Any team game has far more efficient communication when you have voice chat. Heck, even a "casual" MMO like PvE Guild Wars 2 games, have chat-rooms and voice-chats because its way easier to specifically say "My Rocket Boots are charged, can anyone drop a light circle?". And that's not even truly competitive or "hard" compared to FPS games. Go into PvP Guild Wars 2, and you won't get anywhere without voice chat. Coordination is a huge element of the game.
  - munchbunny 8y ago
antisthenes 8y ago
Perhaps I missed it in the article, but did the author control for receiving comments on 'how to play' in winning games vs losing games? Because if not, the whole study seems sort of moot.
tdb7893 8y ago
The hardest thing about this study to me is that these tendencies aren't only applicable to Overwatch. It's obviously worse in video game culture (because it's more abusive and immature as a whole) but this matches with what I've had women engineers tell me of their experiences (especially in college where I know people who got asked out constantly and also were disrespected in group projects)
- 34qg4g 8y ago
  >got asked out constantly oh god the horror
zemo 8y ago
The entire premise of this study is not useful, because what you're asking is "what is the numerical occurrence of quantifiable harassment" and not "what is the rate and which players abandon playing the game as a consequence of feeling unwelcome and how does it vary by gender". What do the numerical rates of harassment tell us, and how are they actionable? If a group of players does not play because they feel unwelcome, THAT is the thing that matters, and your study does absolutely nothing to analyse this criteria. > I avoided ever using voice chat and would call things out in text chat when time permitted. Again, this betrays a deep flaw in methodology, since voice chat is an integral part of competitive Overwatch. The important things to consider here are: what percentage of players are comfortable using voice chat and how does that vary by gender; does availability and usage of voice chat confer any competitive advantage to the player; and do the two factors interact in such a way as to confer a competitive advantage to one gender over the other? I find it disturbing when men do scientific studies to determine if women's stated experiences are real.
- minitech 8y ago
  > If a group of players does not play because they feel unwelcome, THAT is the thing that matters What, in terms of profit? (And in the next paragraph, competitive advantage, really?) That’s a very narrow view of what it means for something to “matter”.
  - zemo 8y ago
    no, in terms of the fact that they literally ran away because of harassment. That's important, and if I have to explain to you why people leaving due to serial harassment is bad for communities, we have nothing to talk about.
    - minitech 8y ago
      We agree that people leaving due to serial harassment is bad for communities. Do you disagree that harassment is an issue even when it doesn’t cause people to leave? Because that’s what it sounds like, and it sounds ridiculous.
dev_dull 8y ago
> For one thing, when I was playing video games in the 90s, a huge fraction of the online gaming population was made up of kids, and online game communities were nicer than they are today Maybe it’s because I grew up playing FPS, but for as far as I remember people in these games were extremely hostile to women.
- ntsplnkv2 8y ago
  Nostalgia is a powerful thing. I will say, though, dedicated servers which were far more common back then typically had admins that would moderate servers. On good servers toxicity was almost nonexistent. I miss those days a lot. I like matchmaking of today a lot too, but some of my best memories came from the communities built on dedicated servers.
  - aidenn0 8y ago
    I think community expectations are important. People in general, (maybe kids especially?) want to fit in, so they will meet whatever the perceived expectations of their peers are.
- dragontamer 8y ago
  I mean, I grew up in the 90s playing "Neopets" and I remember that the online gaming population was made up of kids, but overall more friendly. I also played Starsiege Tribes and Starcraft Brood War online, and... yeah, a bit more toxic over there. The "hardcore competitive" community just oozes toxicity for some reason as far as i remember. But even "Yahoo Chess" (a very peaceful community) would have a difference between my Sister's chat and my chat when we were kids. People really do treat women differently online. My sister was slightly better than me in Chess (she probably won 60% of the time vs me), but otherwise we both were roughly on the same level. The only major difference would be our usernames.
  - crazypyro 8y ago
    Competition can often breed toxicity. I've actually found Starcraft to be less toxic simply because in the most competitive game mode ( 1 versus 1), there's nobody to blame but yourself if you lose. People tend to get toxic about using "cheesy" strategies or playing the "overpowered" race, but overall, the most toxic communities I've been apart of tend to be highly competitive games where team play is a key component. Some examples in my experience are games like Counter-Strike, Dota 2, LoL, etc. A couple common features of these games are: 1. Matches tend to be drawn out over at least 30 minutes, if not longer, forcing you to be teamed with people you may not get along with for a longer period. (Overwatch comes close to having this issue, but I still feel the shorter match times tend to limit extreme toxicity you can find in other games) 2. A single player actively working to harm the team, or simply not at the same skill level as the rest of the players, can easily sink the team's chances of success. 3. An individual matchmaking / visible ranking system which fosters toxic attitudes like "well, I'll sink your rating if I don't like you", since every member of the team is ostensibly trying to raise their rank. #3 when combined with #1 and #2 can create situations where "trolls" can easily ruin a game that may take an hour to finish and may even prevent any chance of victory. These factors tend to be key to the most toxic game communities I've been involved in.
devindotcom 8y ago
>Examples of typical undirected comments that would occur in either condition include ""my girlfriend keeps sexting me how do I get her to stop?", "going in balls deep", "what a surprise. strokes dick [during the post-game highlight]", and "support your local boobies". I think it's notable that although these are not directed they are highly gendered. This is part of the problem as well.
- zucan 8y ago
  I don't see how the examples are any different to gender-neutral sexual comments.
deleted 8y ago
[deleted]
macrael 8y ago
This video series from a gamer who plays with voice chat enabled was pretty eye opening. https://www.youtube.com/watch?v=jL0aVqVslSE https://www.youtube.com/watch?v=jL0aVqVslSE
- 34qg4g 8y ago
  Cherry picked recordings are very eye opening
  - macrael 8y ago
    The series comes out regularly enough to give you all the information you need about the frequency of this shit.
rossdavidh 8y ago
So, it seems like you could probably get a fair number of people to volunteer to participate in a study like this, in order to up the number of games played. Also, I wonder if you could (Laurie Anderson-style) change your voice's gender, so that gender could be manipulated even for games with live voice. If there's a researcher out there who was willing/able to organize, I think there would be a fair number of volunteers.
wowBigDeal 8y ago
Ah yes, very important research about very unimportant passtimes. Advanced button pushing for pixel enthusiasts, and other gender-sensitive activities.
madrox 8y ago
I do wonder if toxicity is inherently built in to the nature of Overwatch. It's a 6v6 team game that incentives you into a play mode that punishes you heavily for leaving a game you aren't enjoying. I realize non-competitive modes are a lot less punishing, but anyone who's played a lot of Overwatch knows if you want to "play Overwatch the way it was meant to be played," you have to go to competitive. Speaking psychologically, I'm curious what that kind of experience does to a person's behavior (short term and long term). I've played hundreds of hours of Overwatch on console with my wife. We don't use mics, because we had enough bad experiences early in our career to make it not worth the risk. I never have problems when I play by myself. Surprising number of people questioning the methodology in this study. I would've expected that if it ran counter to anecdotal experiences described by women (and men who group with them) who play, but I'm not convinced more rigorous methodologies would significantly change the results. If someone does believe that, I'd seriously challenge their reasoning.
- eropple 8y ago
  I don't think toxicity is built in, though I understand where the thinking comes from. I play Overwatch (albeit on PC) and pretty actively use a mic, and I make a point of being the first person to talk in a channel. I'm friendly and I'm even-keeled about it, and that sets the tone. Once that tone is set it's very difficult for your garden-variety douchebag to start mouthing off. The LFG system also helps.
  - JimboOmega 8y ago
    Comparing overwatch to League of Legends, League is sooo much worse, and I don't know why. Also FWIW the voice you use in OW can cause drama. I've definitely had games where I get flak for my voice - when it was more androgynous often people would want to know my gender. I've seen people get exploded on for their accents, sounding young, etc.
    - eropple 8y ago
      Yeah, that can make sense. I'm a white American male with a deep voice, people aren't going to give me much shit. But by being friendly I can make things easier for other folks, and I occasionally get PMs from people who tell me they appreciate that.
MobileVet 8y ago
Lots of one off gender swaps that have been shared and showed similar bias. Eg https://blog.olark.com/live-chat-gender-equality-experiment https://blog.olark.com/live-chat-gender-equality-experiment
LowLevelHacker 8y ago
A man was playing both roles so the outcome makes no sense. You would need to randomize withing women and within men. Having a man play a woman means nothing.
jordanthoms 8y ago
If I took up soccer, I'd be unlikely to join a team with some 8 year olds - even though I'd probably be at their skill level. Instead, I'd find a group of people who I'd get along with better. So why do games do matchmaking based only on skill, not taking social cohesion into account? There's much more to a good gaming experience than just having skill-balanced teams. CS:GO is a classic case of this - the player base is noticeably more mature at certain times of day, and there's a distinct period when all the kids have just got home from school where things are particularly bad. Occasionally you'll get a team where you everyone works together well, but to have that every time you'll need to spend a lot of effort maintaining a huge friends list in order to be able to put a team together at any given time - rather than just pressing the matchmaking button. (I think you could might get decent results on this by having each player rate the others at the end of the match - but you don't use the ratings to rank the players, instead you'd use them to cluster players based on who they mutually like to play with. You'd then use that as a variable to optimize on when doing matchmaking. Hopefully with enough data you'd be able to predict if players would have a good experience even if they've never played together before.)
- nordsieck 8y ago
  > I think you could might get decent results on this by having each player rate the others at the end of the match - but you don't use the ratings to rank the players, instead you'd use them to cluster players based on who they mutually like to play with. Dota has a system like this called "behavior score". I know it works based on reports, but it might include other things like text analysis or item feeding. It seems to work pretty well.
- AnIdiotOnTheNet 8y ago
  Since matchmaking has taken over, I basically don't even play online multiplayer games anymore. I feel fortunate to have grown up in a time where there were dedicated server communities. It's a bit like Cheers, sometimes you wanna go where everybody knows your name and you don't really mind that they routinely wipe the floor with you.
- Mirioron 8y ago
  >So why do games do matchmaking based only on skill, not taking social cohesion into account? There's much more to a good gaming experience than just having skill-balanced teams. Not enough players. Any matchmaking system requires a large number of players to be playing at once to work. People are generally unwilling to wait several minutes or more to get into a game. The problem is that this has a cascading effect: the longer your queue times the more people will quit and the longer your queue gets. Below a certain threshold the system breaks down. If you start classifying people based on criteria such as social cohesion then you increase matchmaking time even further. There are some games that could have a matchmaking system like this. But no developer knows in the development stage that their game is going to have this kind of popularity, which means that this type of matchmaking would have to be built after the fact. At that point you risk ruining something people like about the game. Another thing to keep in mind is that player numbers fluctuate significantly. The number of players playing at 5 am between Monday and Tuesday is much lower than the number of players playing at 7 pm on a Friday evening. You would have to change matchmaking algorithms based on time of day or activity. You would also have to change matchmaking algorithms based on skill level. At the very top end matchmaking is going to be long due to there simply not being enough players at that rating. This can have a negative impact on the retention of players, because popular streamers usually have a high rating. Watching a stream of matchmaking queue isn't very interesting and doesn't make people want to play the game. Then there's the question of whether you even should do this. People make friends in games. If a game introduces matchmaking on social cohesion you essentially put players into a filter bubble. We see from Facebook and Reddit that these kinds of filter bubbles aren't ideal. I think one of the coolest things that I've got from games is that I've interacted with and made friends with people wildly different from me.
gpm 8y ago
Lots of comments here about voice chat. I wonder if there is software that can convincingly change the gender of a voice in real time? Could make for an interesting experiment.

202 points by Aissen 8y ago

This is about as well done of a study as we could ask for from a non-academic research level. Basically while the number of sexual comments were relatively low, the amount of questioning/challenging of his abilities went up a lot when he played with a feminine name. Reminds me of a professor at Stanford who underwent a sex change and experienced very different interactions from his peers: https://www.nature.com/scitable/content/does-gender-matter-by-ben-a-barres-10602856 https://www.nature.com/scitable/content/does-gender-matter-b... We recently did a survey of 240 collegiate esports players and found that women were 9x more likely to be "very concerned" about sexism and 2x as likely to be "very concerned" about bullying in competitive gaming culture. I think Dan's research can help explain why. https://www.midgame.gg/survey https://www.midgame.gg/survey

ToFundorNot 8y ago
This study is missing one important factor, a baseline. The author should have provided an androgynous named account to see how that fared.
- Dylan16807 8y ago
  An A/B test doesn't need to attempt a "middle" option. And there's no reason to assume an androgynous name would be in the middle or in any way qualify as a "baseline".
  - zaarn 8y ago
    It would still be very interesting to see how people would react to neutral usernames like randomly generated number sequences. It could provide some useful insight.
    - seqastian 8y ago
      Gamers just default to male.
      - diegoperini 8y ago
        Depends on the game but I see your point.
      - zaarn 8y ago
        That's just an assertion without data backing it up. The above experiment with neutral names would give some insight if "gamers" do indeed "default to male", as you express it, or not.
towaway1138 8y ago
> about as well done of a study as we could ask for from a non-academic research level Unfortunately, there's a good reason why academics use proper experimental design. Without it, all you really have is suggestive noise, if even that.
- macleginn 8y ago
  The definition of “proper experimental design” varies widely by the field, and DL’s methodology looks quite solid, as far as social sciences and the task at hand are concerned.
  - leereeves 8y ago
    The generally poor quality of research in the social sciences (proven so by the lack of reproducibility) is not really a good defense for the poor quality of any individual study.
    - naasking 8y ago
      The author is pretty rigourous about explaining the limitations.
    - esailija 8y ago
      Yes, it is? Science does not guarantee knowledge in any field, it's only supposed to increase the odds. Are the odds of getting true knowledge in this field increased compared to the alternatives (I.E. people just arbitrarily claiming stuff based on personal anecdotes?) Then it's the best we have in that field and rationally you have no reason to believe the claims of knowledge acquired from worse methods over the better alternatives - just like it works in other fields.
      - leereeves 8y ago
        Why do you assume research in the social sciences even increases the odds of a true result? Combining low standards with a publish or perish culture actually encourages the publication of spurious results (and worse behavior) and hinders the careers of researchers with higher standards.

Nexxxeh 8y ago

Really interesting read. I appreciate that it highlights the problems with the data and doesn't reach ridiculous conclusions based on junk data. Anecdotally I've found a noticeable improvement since Endorsements, which this data missed. Also the rise of Fortnite and more recently Apex Legends has led to a further decrease in toxicity that I've seen. A lot of toxic kids left for BR pastures. That's on PC in the EU region on a high-silver current, low-gold peak SR account, casual QP-heavy play but some comp, pure flexing (as I prefer to fill rather than focus on maining one or two characters. Maining is arguably the best way to climb out to Plat). I'm an almost ridiculously positive player with a gender neutral name and I've found being overtly positive vastly improves the experience. It does prompt a couple of toxic responses ("kys", "fuck off" etc) but the majority is really positive. (I generally just report any toxicity.) Playing without voice in Comp is sub-optimal, as winning is much easier with some kind of synergy. Even a basic PharMercy combo, or ultimate combos like a EMP+DVa Bomb can be highly effective. I normally Solo Q, but playing paired with a friend in QP yesterday I had a surreal experience. One player on the enemy team insulted my friend (sexual but not specifically-gendered insult). She was playing with a non-gendered name, as Moira, and the toxic enemy player complained she was carrying us. All the players on both teams turned on the toxic player, calling them out and highlighting why they were actually losing. I have not seen anything like it before but it was hilarious. Stay As A Team is great in theory, but a team of 6 randomised will get rekt by a pre-made 6 stack on Discord with each other, and the matchmaker doesn't seem to take that into account. I wonder what region author was playing on?

kfwhp 8y ago
Every time I see a feminine name in a videogame I think the person is just fishing for attention. This is important because most players of both genders use neutral-looking names. The study doesn't account for this.
- SolaceQuantum 8y ago
  Honestly, I don't really see such behavior directed towards male names though. Stuff like LucasJones or MikeAtLarge or LordOfKilling doesn't get the same attention that SuzieHealsU or CuteLamb or PrincessHurtful might. If it was equal, I'd expect 'MrKickass' to be questioned and assumed to be grabbing attntion that 'MsKilljoy' gets... but in my experience this distinctly isn't the case.

ggggtez 8y ago

Obviously, a number of issues present in the study, but I can appreciate they tried to account for some of them. Lots of variables are unaccounted for like time of day, win/loss, character, what the researcher said in chat, etc. Similarly, the opening preamble points out that often the worst harassment is potentially related to when players hear female voices. Obviously there is a different way you could do this study, which would be to invite ~300 players of various skill levels to play a single game and record the results for analysis. That type of study would have it's own problems, but you could at least be more likely to control for some of the problems in this study.

mactrey 8y ago
Yeah, the voice is the key. The gender ratio is skewed so much at higher ranks that an obviously feminine name will simply be assumed to be ironic. I would imagine the toxicity/sexual comments would increase by an order of magnitude if the voice was female-presenting along with the account name.
novia 8y ago
Or this researcher could have got a woman to speak into his mic while he played.

AcerbicZero 8y ago

This is an interesting study, and helps highlight why Overwatch needs a scoreboard. With 6 random people on a team, very limited personal statistics, and no game length team statistics, everyone tends to get toxic once you start losing. I've played everything from high gold (tanked rank for a few seasons) to low masters, and its the same pattern every time. Someone makes an accusation about a player not pulling their weight, demands a swap in a rude way, and the player being accused defends themselves. Then the name calling starts. It would be interesting to see what roles the author was playing, and the exact SR of the accounts, as there seems to be a great deal of variation in what people will say on voice comms (usually higher SR, because voice comms > chat) and what will be said in chat. People also like to blame DPS first, then demand a Reinhardt (if they don't have one) and then finally blame support. Without hard data as to what is going wrong with a team composition, toxicity is almost guaranteed when losing.

quxbar 8y ago
To be fair to the paper, toxicity is almost guaranteed when losing while playing with a feminine name. It's pretty rare otherwise.
wlesieutre 8y ago
> People also like to blame DPS first, then demand a Reinhardt (if they don't have one) and then finally blame support. As someone who mains support, I don't think this is true. I've had plenty of games where I do a shitton of healing, get the healing card at the end, and still had people blaming me for "0 heals" all game. A scoreboard would help with that, but it'd be a trade-off with having 5 angry people shitting on you if you ever have a bad game. I'm not sure that's better. EDIT: Plus there's all the times when your stats are bad but you're not the right one to blame, like when the whole team can't move up through a choke because the tanks are too chicken to go first, but it gets reflected in poor DPS eliminations. Ultimately it's a team game, and failures of teamwork aren't reflected well in a stats scoreboard. I think you'd incentivize poor team play by people trying to pad their own numbers at the team's expense. And no matter what information is available, some people are just looking for someone to blame when they lose. I'm not sure how to fix that. If matchmaking is working right you're going to lose about half the time, there's no point in getting worked up about it but people still do.

NullPrefix 8y ago

>There are a lot of games that weren't included in the experiment because I wasn't in a mood to listen to someone rage at their team for fifteen minutes and the procedure I used involved pre-committing to not muting people who do that.

SolaceQuantum 8y ago

"If that result generalizes to OW, that would explain something which I thought was odd, which was that a lot of demands to switch and general vitriol came during my best performances with the feminine account. " This isn't statistically analyzed, but I think it's a fairly significant detail that's not emphasized more in the paper.

qiqing 8y ago
Agreed. Especially this excerpt: > Per above, I didn't want to code tone of messages to avoid bias, so this table only indicates the rate at which people told me I was playing incorrectly or asked that I switch to a different character. The qualitative difference in experience is understated by this table. For example, the one time someone asked me to switch characters in the masculine condition, the request was a one sentence, polite, request ("hey, we're dying too quickly, could we switch [from the standard one primary healer / one off healer setup] to double primary healer or switch our tank to [a tank that can block more damage]?"). When using the feminine name, a typical case would involve 1-4 people calling me human garbage for most of the game and consoling themselves with the idea that the entire reason our team is losing is that I won't change characters.

nickpsecurity 8y ago

Great study! Only gripe is that most trolling I get in online gaming is through voice chat. They also seem to troll me a tad bit more if I have a mic plugged in but I'm not talking. I suggest a re-run of this experiment recording the audio in case we get wildly different results. In a one-party consent state if you want to play it safe. ;) I also think it will vary game by game. I don't play a lot of games. Most are shooters like Modern Warfare, Battlefield, and Mass Effect 3. Modern Warfare 2 and 3 on Xbox brought in more trolls in voice chat than all my others combined except maybe GTA V. In the last one, you play as a vicious thug causing problems for everyone around you. Them being pricks makes more sense. The difference in shooter games, though, was more interesting given it might reflect different sub-cultures in gaming. Specifically, MW is designed to appeal to largest number of people with lowest barrier to entry and patience required. Might self-select for higher percentage of jerks. Whereas, Battlefield 4 and Mass Effect 3 push people to work together more for best results.

towaway1138 8y ago
I don't understand why people ever play with voice chat enabled. I don't want to hear streams of ungrammatical foul language, and I certainly don't want to hear people slapping their kids, etc. TMI.
- jdietrich 8y ago
  It's absolutely necessary if you want to perform well in a team-based game.
- dragontamer 8y ago
  Any team game has far more efficient communication when you have voice chat. Heck, even a "casual" MMO like PvE Guild Wars 2 games, have chat-rooms and voice-chats because its way easier to specifically say "My Rocket Boots are charged, can anyone drop a light circle?". And that's not even truly competitive or "hard" compared to FPS games. Go into PvP Guild Wars 2, and you won't get anywhere without voice chat. Coordination is a huge element of the game.
- munchbunny 8y ago

antisthenes 8y ago

Perhaps I missed it in the article, but did the author control for receiving comments on 'how to play' in winning games vs losing games? Because if not, the whole study seems sort of moot.

tdb7893 8y ago

The hardest thing about this study to me is that these tendencies aren't only applicable to Overwatch. It's obviously worse in video game culture (because it's more abusive and immature as a whole) but this matches with what I've had women engineers tell me of their experiences (especially in college where I know people who got asked out constantly and also were disrespected in group projects)

34qg4g 8y ago
>got asked out constantly oh god the horror

zemo 8y ago

The entire premise of this study is not useful, because what you're asking is "what is the numerical occurrence of quantifiable harassment" and not "what is the rate and which players abandon playing the game as a consequence of feeling unwelcome and how does it vary by gender". What do the numerical rates of harassment tell us, and how are they actionable? If a group of players does not play because they feel unwelcome, THAT is the thing that matters, and your study does absolutely nothing to analyse this criteria. > I avoided ever using voice chat and would call things out in text chat when time permitted. Again, this betrays a deep flaw in methodology, since voice chat is an integral part of competitive Overwatch. The important things to consider here are: what percentage of players are comfortable using voice chat and how does that vary by gender; does availability and usage of voice chat confer any competitive advantage to the player; and do the two factors interact in such a way as to confer a competitive advantage to one gender over the other? I find it disturbing when men do scientific studies to determine if women's stated experiences are real.

minitech 8y ago
> If a group of players does not play because they feel unwelcome, THAT is the thing that matters What, in terms of profit? (And in the next paragraph, competitive advantage, really?) That’s a very narrow view of what it means for something to “matter”.
- zemo 8y ago
  no, in terms of the fact that they literally ran away because of harassment. That's important, and if I have to explain to you why people leaving due to serial harassment is bad for communities, we have nothing to talk about.
  - minitech 8y ago
    We agree that people leaving due to serial harassment is bad for communities. Do you disagree that harassment is an issue even when it doesn’t cause people to leave? Because that’s what it sounds like, and it sounds ridiculous.

dev_dull 8y ago

> For one thing, when I was playing video games in the 90s, a huge fraction of the online gaming population was made up of kids, and online game communities were nicer than they are today Maybe it’s because I grew up playing FPS, but for as far as I remember people in these games were extremely hostile to women.

ntsplnkv2 8y ago
Nostalgia is a powerful thing. I will say, though, dedicated servers which were far more common back then typically had admins that would moderate servers. On good servers toxicity was almost nonexistent. I miss those days a lot. I like matchmaking of today a lot too, but some of my best memories came from the communities built on dedicated servers.
- aidenn0 8y ago
  I think community expectations are important. People in general, (maybe kids especially?) want to fit in, so they will meet whatever the perceived expectations of their peers are.
dragontamer 8y ago
I mean, I grew up in the 90s playing "Neopets" and I remember that the online gaming population was made up of kids, but overall more friendly. I also played Starsiege Tribes and Starcraft Brood War online, and... yeah, a bit more toxic over there. The "hardcore competitive" community just oozes toxicity for some reason as far as i remember. But even "Yahoo Chess" (a very peaceful community) would have a difference between my Sister's chat and my chat when we were kids. People really do treat women differently online. My sister was slightly better than me in Chess (she probably won 60% of the time vs me), but otherwise we both were roughly on the same level. The only major difference would be our usernames.
- crazypyro 8y ago
  Competition can often breed toxicity. I've actually found Starcraft to be less toxic simply because in the most competitive game mode ( 1 versus 1), there's nobody to blame but yourself if you lose. People tend to get toxic about using "cheesy" strategies or playing the "overpowered" race, but overall, the most toxic communities I've been apart of tend to be highly competitive games where team play is a key component. Some examples in my experience are games like Counter-Strike, Dota 2, LoL, etc. A couple common features of these games are: 1. Matches tend to be drawn out over at least 30 minutes, if not longer, forcing you to be teamed with people you may not get along with for a longer period. (Overwatch comes close to having this issue, but I still feel the shorter match times tend to limit extreme toxicity you can find in other games) 2. A single player actively working to harm the team, or simply not at the same skill level as the rest of the players, can easily sink the team's chances of success. 3. An individual matchmaking / visible ranking system which fosters toxic attitudes like "well, I'll sink your rating if I don't like you", since every member of the team is ostensibly trying to raise their rank. #3 when combined with #1 and #2 can create situations where "trolls" can easily ruin a game that may take an hour to finish and may even prevent any chance of victory. These factors tend to be key to the most toxic game communities I've been involved in.

devindotcom 8y ago

>Examples of typical undirected comments that would occur in either condition include ""my girlfriend keeps sexting me how do I get her to stop?", "going in balls deep", "what a surprise. strokes dick [during the post-game highlight]", and "support your local boobies". I think it's notable that although these are not directed they are highly gendered. This is part of the problem as well.

zucan 8y ago
I don't see how the examples are any different to gender-neutral sexual comments.

deleted 8y ago

[deleted]

macrael 8y ago

This video series from a gamer who plays with voice chat enabled was pretty eye opening. https://www.youtube.com/watch?v=jL0aVqVslSE https://www.youtube.com/watch?v=jL0aVqVslSE

34qg4g 8y ago
Cherry picked recordings are very eye opening
- macrael 8y ago
  The series comes out regularly enough to give you all the information you need about the frequency of this shit.

rossdavidh 8y ago

So, it seems like you could probably get a fair number of people to volunteer to participate in a study like this, in order to up the number of games played. Also, I wonder if you could (Laurie Anderson-style) change your voice's gender, so that gender could be manipulated even for games with live voice. If there's a researcher out there who was willing/able to organize, I think there would be a fair number of volunteers.

wowBigDeal 8y ago

Ah yes, very important research about very unimportant passtimes. Advanced button pushing for pixel enthusiasts, and other gender-sensitive activities.

madrox 8y ago

I do wonder if toxicity is inherently built in to the nature of Overwatch. It's a 6v6 team game that incentives you into a play mode that punishes you heavily for leaving a game you aren't enjoying. I realize non-competitive modes are a lot less punishing, but anyone who's played a lot of Overwatch knows if you want to "play Overwatch the way it was meant to be played," you have to go to competitive. Speaking psychologically, I'm curious what that kind of experience does to a person's behavior (short term and long term). I've played hundreds of hours of Overwatch on console with my wife. We don't use mics, because we had enough bad experiences early in our career to make it not worth the risk. I never have problems when I play by myself. Surprising number of people questioning the methodology in this study. I would've expected that if it ran counter to anecdotal experiences described by women (and men who group with them) who play, but I'm not convinced more rigorous methodologies would significantly change the results. If someone does believe that, I'd seriously challenge their reasoning.

eropple 8y ago
I don't think toxicity is built in, though I understand where the thinking comes from. I play Overwatch (albeit on PC) and pretty actively use a mic, and I make a point of being the first person to talk in a channel. I'm friendly and I'm even-keeled about it, and that sets the tone. Once that tone is set it's very difficult for your garden-variety douchebag to start mouthing off. The LFG system also helps.
- JimboOmega 8y ago
  Comparing overwatch to League of Legends, League is sooo much worse, and I don't know why. Also FWIW the voice you use in OW can cause drama. I've definitely had games where I get flak for my voice - when it was more androgynous often people would want to know my gender. I've seen people get exploded on for their accents, sounding young, etc.
  - eropple 8y ago
    Yeah, that can make sense. I'm a white American male with a deep voice, people aren't going to give me much shit. But by being friendly I can make things easier for other folks, and I occasionally get PMs from people who tell me they appreciate that.

MobileVet 8y ago

Lots of one off gender swaps that have been shared and showed similar bias. Eg https://blog.olark.com/live-chat-gender-equality-experiment https://blog.olark.com/live-chat-gender-equality-experiment

LowLevelHacker 8y ago

A man was playing both roles so the outcome makes no sense. You would need to randomize withing women and within men. Having a man play a woman means nothing.

jordanthoms 8y ago

If I took up soccer, I'd be unlikely to join a team with some 8 year olds - even though I'd probably be at their skill level. Instead, I'd find a group of people who I'd get along with better. So why do games do matchmaking based only on skill, not taking social cohesion into account? There's much more to a good gaming experience than just having skill-balanced teams. CS:GO is a classic case of this - the player base is noticeably more mature at certain times of day, and there's a distinct period when all the kids have just got home from school where things are particularly bad. Occasionally you'll get a team where you everyone works together well, but to have that every time you'll need to spend a lot of effort maintaining a huge friends list in order to be able to put a team together at any given time - rather than just pressing the matchmaking button. (I think you could might get decent results on this by having each player rate the others at the end of the match - but you don't use the ratings to rank the players, instead you'd use them to cluster players based on who they mutually like to play with. You'd then use that as a variable to optimize on when doing matchmaking. Hopefully with enough data you'd be able to predict if players would have a good experience even if they've never played together before.)

nordsieck 8y ago
> I think you could might get decent results on this by having each player rate the others at the end of the match - but you don't use the ratings to rank the players, instead you'd use them to cluster players based on who they mutually like to play with. Dota has a system like this called "behavior score". I know it works based on reports, but it might include other things like text analysis or item feeding. It seems to work pretty well.
AnIdiotOnTheNet 8y ago
Since matchmaking has taken over, I basically don't even play online multiplayer games anymore. I feel fortunate to have grown up in a time where there were dedicated server communities. It's a bit like Cheers, sometimes you wanna go where everybody knows your name and you don't really mind that they routinely wipe the floor with you.
Mirioron 8y ago
>So why do games do matchmaking based only on skill, not taking social cohesion into account? There's much more to a good gaming experience than just having skill-balanced teams. Not enough players. Any matchmaking system requires a large number of players to be playing at once to work. People are generally unwilling to wait several minutes or more to get into a game. The problem is that this has a cascading effect: the longer your queue times the more people will quit and the longer your queue gets. Below a certain threshold the system breaks down. If you start classifying people based on criteria such as social cohesion then you increase matchmaking time even further. There are some games that could have a matchmaking system like this. But no developer knows in the development stage that their game is going to have this kind of popularity, which means that this type of matchmaking would have to be built after the fact. At that point you risk ruining something people like about the game. Another thing to keep in mind is that player numbers fluctuate significantly. The number of players playing at 5 am between Monday and Tuesday is much lower than the number of players playing at 7 pm on a Friday evening. You would have to change matchmaking algorithms based on time of day or activity. You would also have to change matchmaking algorithms based on skill level. At the very top end matchmaking is going to be long due to there simply not being enough players at that rating. This can have a negative impact on the retention of players, because popular streamers usually have a high rating. Watching a stream of matchmaking queue isn't very interesting and doesn't make people want to play the game. Then there's the question of whether you even should do this. People make friends in games. If a game introduces matchmaking on social cohesion you essentially put players into a filter bubble. We see from Facebook and Reddit that these kinds of filter bubbles aren't ideal. I think one of the coolest things that I've got from games is that I've interacted with and made friends with people wildly different from me.

gpm 8y ago

Lots of comments here about voice chat. I wonder if there is software that can convincingly change the gender of a voice in real time? Could make for an interesting experiment.