4 ms·

Just because there are false positives or one company has a bad implementation doesn't make it an unsolveable problem. As parent mentioned, ml sentiment analys

by ethbro 8y ago

Just because there are false positives or one company has a bad implementation doesn't make it an unsolveable problem. As parent mentioned, ml sentiment analysis + logging + peer reporting + final human analysis = problem effectively combated It's the final human analysis that companies are loathe to fund (e.g. Facebook), as costs scale with user count. But it's not efficiently unsolveable with current tech. Therefore, companies simply aren't prioritizing it. And won't, as long as the impact of toxic users isn't impacting the bottom line.

shawn 8y ago
Kay. What are the odds that some HN commenters are going to solve the same problem that has been an active priority for years of the people in the field? I can’t speak for Valve or Riot, but at S2 it was a concerning issue. There’s just no good way to do it when the people involved are actively malicious. If you think there is, get ready to have your community collapse around you as everyone complains about unfair bans. I don’t think you really appreciate the scale of the problem. Final human analysis is not possible when there are literally millions of games per week. It’s also not something that ML can identify cleanly — the moment it does, the culture will adapt to bypass the evaluator. It always does.
- ethbro 8y ago
  I absolutely appreciate the scale of the problem, and the adversarial nature of peer reporting. My day job has those same characteristics. Millions of games per week * 30 minutes per game * avg lines of chat per minute = manageable w/ a proper streaming architecture Especially when you have access to a massive, perfectly-scaled, distributed edge compute system. (i.e. running minimal, performance-optimized models on users' opponents' clients to do the initial detection / filter / compression pass) But my point is this is fundamentally an economic problem, given current state of the art, not a technical one. Companies are looking for pure-technical solutions because they're cheaper, and then complaining that it's a hard problem because they're unwilling to properly fund hybrid systems until state of the art can deliver. ML is a first order approximation of human ability, not a magic unicorn that gives you exactly what you want. Thats the definitions of engineering: how do I build a system that fulfills my requirements from the pieces I have, not the pieces I wish I had? So I don't feel much pity when companies allow toxic user bases to flourish because it's cheaper than funding solutions. * Above intended in no way to belittle the awesome work folks are doing in the space with ML detection. But sometimes as engineers we need to admit when management is making unethical choices for financial gain
  - shawn 8y ago
    You're quite cocky. That's not an insult. I think if you cut your teeth on this problem for a couple months, you'd look back on this comment with some mixed feelings. When a system is inherently toxic, there is no way to make it not-toxic short of coming up with a different substance. We can't make nuclear waste pure just because we wish it were. This is a very similar situation to this particular type of game. When it's a team game, and you rely on people who aren't doing their job, and you're separated by distance, it brings out the worst in you. The problem isn't chat. If it were, it would be solved already, for the reasons you point out. The problem is the people. When someone doesn't like someone else, they will find a way to ruin their game while bypassing your censors. And yes, you can ban some of them, but not most of them, and not when most of the community acts like this. Which they do. Which you can't understand unless you go play the game for a bit.
    - ethbro 8y ago
      Toxicity is learned and reinforced (or not) behavior. In the same way that HN comments are generally pretty constructive and charitable towards each other (and substantially more so than similar forums), so does negativity spread. I'm not expecting any company to be able to turn their community into a paragon of empathy overnight. But I do believe that putting systems in place that punish poor behavior and reward good behavior has an impact in the long run. From my first comment: > Therefore, companies simply aren't prioritizing [addressing the problem]. And won't, as long as the impact of toxic users isn't impacting the bottom line. Because I honestly don't believe they care, in any non-monetary sense. They're essentially amoral. If they can make a billion dollars while people shout racist or sexist slurs at each other, they're fine with that. And that's the primary annoyance I have with game companies. We hear "The player base is caustic, and we can't do anything about it, because it's too hard." When I believe reality is closer to "We ignored this problem as an industry (boy will be boys!), and now we don't want to be blamed for helping raise a generation of borderline sociopaths." And yes, I spent many a college evening watching friends troll internet strangers in DotA, pre-LoL. So I'm familiar with the concept and execution.
      - 8y ago

by ethbro 8y ago

shawn 8y ago

Kay. What are the odds that some HN commenters are going to solve the same problem that has been an active priority for years of the people in the field? I can’t speak for Valve or Riot, but at S2 it was a concerning issue. There’s just no good way to do it when the people involved are actively malicious. If you think there is, get ready to have your community collapse around you as everyone complains about unfair bans. I don’t think you really appreciate the scale of the problem. Final human analysis is not possible when there are literally millions of games per week. It’s also not something that ML can identify cleanly — the moment it does, the culture will adapt to bypass the evaluator. It always does.

ethbro 8y ago
I absolutely appreciate the scale of the problem, and the adversarial nature of peer reporting. My day job has those same characteristics. Millions of games per week * 30 minutes per game * avg lines of chat per minute = manageable w/ a proper streaming architecture Especially when you have access to a massive, perfectly-scaled, distributed edge compute system. (i.e. running minimal, performance-optimized models on users' opponents' clients to do the initial detection / filter / compression pass) But my point is this is fundamentally an economic problem, given current state of the art, not a technical one. Companies are looking for pure-technical solutions because they're cheaper, and then complaining that it's a hard problem because they're unwilling to properly fund hybrid systems until state of the art can deliver. ML is a first order approximation of human ability, not a magic unicorn that gives you exactly what you want. Thats the definitions of engineering: how do I build a system that fulfills my requirements from the pieces I have, not the pieces I wish I had? So I don't feel much pity when companies allow toxic user bases to flourish because it's cheaper than funding solutions. * Above intended in no way to belittle the awesome work folks are doing in the space with ML detection. But sometimes as engineers we need to admit when management is making unethical choices for financial gain
- shawn 8y ago
  You're quite cocky. That's not an insult. I think if you cut your teeth on this problem for a couple months, you'd look back on this comment with some mixed feelings. When a system is inherently toxic, there is no way to make it not-toxic short of coming up with a different substance. We can't make nuclear waste pure just because we wish it were. This is a very similar situation to this particular type of game. When it's a team game, and you rely on people who aren't doing their job, and you're separated by distance, it brings out the worst in you. The problem isn't chat. If it were, it would be solved already, for the reasons you point out. The problem is the people. When someone doesn't like someone else, they will find a way to ruin their game while bypassing your censors. And yes, you can ban some of them, but not most of them, and not when most of the community acts like this. Which they do. Which you can't understand unless you go play the game for a bit.
  - ethbro 8y ago
    Toxicity is learned and reinforced (or not) behavior. In the same way that HN comments are generally pretty constructive and charitable towards each other (and substantially more so than similar forums), so does negativity spread. I'm not expecting any company to be able to turn their community into a paragon of empathy overnight. But I do believe that putting systems in place that punish poor behavior and reward good behavior has an impact in the long run. From my first comment: > Therefore, companies simply aren't prioritizing [addressing the problem]. And won't, as long as the impact of toxic users isn't impacting the bottom line. Because I honestly don't believe they care, in any non-monetary sense. They're essentially amoral. If they can make a billion dollars while people shout racist or sexist slurs at each other, they're fine with that. And that's the primary annoyance I have with game companies. We hear "The player base is caustic, and we can't do anything about it, because it's too hard." When I believe reality is closer to "We ignored this problem as an industry (boy will be boys!), and now we don't want to be blamed for helping raise a generation of borderline sociopaths." And yes, I spent many a college evening watching friends troll internet strangers in DotA, pre-LoL. So I'm familiar with the concept and execution.
    - 8y ago