4 ms·

Actually our AUC is 90% and that beats the state of the art in the literature. Accuracy is easy to game. We could get 97.5% accuracy by just predicting False

Actually our AUC is 90% and that beats the state of the art in the literature. Accuracy is easy to game. We could get 97.5% accuracy by just predicting False all of the time. We don't advise reviewers to use the prediction directly, but rather to set thresholds on the probability. 70% probability is a very conservative filter and will catch about 10% and a little bit less than half of them will actually be damaging. So, this is useful enough to reduce the workload by 90% right now. As we increase the model's fitness, we should be able to drive that up to 95%, but I doubt we'll get much better than that.

jacquesm 11y ago
> 70% probability is a very conservative filter and will catch about 10% and a little bit less than half of them will actually be damaging. Right. This is really interesting. A while ago I wrote a classifier that predicted based on a number of criteria if a new member of a web community would be trouble or not. It also worked as an aid in classification and not as a stand-alone decision maker. You have a lot more data to work with so that should make the problem a bit more tractable. Does it use other bits of knowledge about the edit prior probability of a revert for an edit by that particular user or network range for instance?