2 ms·
I don't think this technique is going to be all that helpful. This is because you cannot consider specificity and sensitivity without taking into account the pr
by RcouF1uZ4gsC 12y ago
I don't think this technique is going to be all that helpful. This is because you cannot consider specificity and sensitivity without taking into account the prevalence. Let us look at the data from the paper [1].
From the chart on page 3, we can see that the percentage of banned ranges from 1.7% to 3.3%. On the last page we have the phrase "Though average classifier precision is relatively high (0.80), one in five users identified as antisocial are nonetheless misclassified". Based on this, I think we have a specificity of 80%, and a sensitivity of 80%.
Let us look at 1000 hypothetical users, using 3% as the prevalence. Flagged and Not Flagged represent the output of the algorithm, and Regular and Banned represent "Gold Standard"
|Regular | Banned |
--------------------------------
Total |970 | 30
----------------------------------
Flagged |194 | 24
----------------------------------
Not Flagged | 776 | 6
In this example, the algorithm will flag 218 out of a thousand people as being trolls. 194 of the 218 people flagged are actually not trolls. Doing any sort of automated banning based on this algorithm will I think be unacceptable.
I may have some of my stats or assumptions wrong, and I would love to be educated why if that is the case.
1. http://arxiv.org/pdf/1504.00680v1.pdf http://arxiv.org/pdf/1504.00680v1.pdf