3 ms·

Couldn't this have a high risk of overfitting (not sure if that's exactly the right word, but I think it conveys the idea) when using it as a predictive or proa

Couldn't this have a high risk of overfitting (not sure if that's exactly the right word, but I think it conveys the idea) when using it as a predictive or proactive measure? Just because you observe a correlation in the past doesn't mean it will continue into the future, and there's a lot resting on this prediction.

macintux 10y ago
It sounds like Riot handled this very well: rather than just fire people based on their analysis, they sat down with the employees to discuss the matter. So, yes, making an irrevocable decision based on one ambiguous data point is a bad idea, but using it as an indicator that a conversation should take place strikes me as wise.
drabiega 10y ago
That's a really good reason for not making decisions outright based on the information. You'd want to use the results of these models as a means to direct attention to where it is most likely to be needed. It sounds like this is exactly what they did. Of course, the other side is that the chat data might create biases. The worry here would be that reading the chat logs might create a unfavorable opinion which then leads to unfair interpretations of workplace behaviors. If I was implementing such an action, I would make sure that the people checking in on those flagged by the system don't actually have access to the chatlogs, except in egregious situations.
selectron 10y ago
While overfitting is always possible, in this situation there is an easy causal explanation for why toxic players could be toxic employees. From my experience playing Dota I know that if I was stressed out by work or life I would be more toxic in game. In general another way to phrase it is that correlations in the past strongly suggest correlations in the future. This is especially true when you can think of reasons the variables are causally related. If they did this study and found that employees whose favorite cereal was frosted flakes are more toxic I'd be much more skeptical, as that could indicate they were doing a shoot everywhere and see what sticks approach.

macintux 10y ago

It sounds like Riot handled this very well: rather than just fire people based on their analysis, they sat down with the employees to discuss the matter. So, yes, making an irrevocable decision based on one ambiguous data point is a bad idea, but using it as an indicator that a conversation should take place strikes me as wise.

drabiega 10y ago

That's a really good reason for not making decisions outright based on the information. You'd want to use the results of these models as a means to direct attention to where it is most likely to be needed. It sounds like this is exactly what they did. Of course, the other side is that the chat data might create biases. The worry here would be that reading the chat logs might create a unfavorable opinion which then leads to unfair interpretations of workplace behaviors. If I was implementing such an action, I would make sure that the people checking in on those flagged by the system don't actually have access to the chatlogs, except in egregious situations.

selectron 10y ago

While overfitting is always possible, in this situation there is an easy causal explanation for why toxic players could be toxic employees. From my experience playing Dota I know that if I was stressed out by work or life I would be more toxic in game. In general another way to phrase it is that correlations in the past strongly suggest correlations in the future. This is especially true when you can think of reasons the variables are causally related. If they did this study and found that employees whose favorite cereal was frosted flakes are more toxic I'd be much more skeptical, as that could indicate they were doing a shoot everywhere and see what sticks approach.