3 ms·
Results in this field usually look way better than they would be in a production environment. Regarding articles you mentioned: Using ROC curve for evaluation
by Scea91 9y ago
Results in this field usually look way better than they would be in a production environment.
Regarding articles you mentioned:
Using ROC curve for evaluation in this case is a red-flag because it doesn't take the data imbalance into account. Precision-Recall curve is way more suitable. You can have great AUC on the ROC curve but precision can be near zero in highly imbalanced problems such as malware detection. Precision is the probability that a positive detection is a true positive. Which is usually the measure you are most interested in.
The problem is that precision changes if you change the class priors. Because of that, the results are always very dataset specific.
With that said, I do not say that machine learning or neural networks do not work on this task. It just doesn't work in the end-to-end manner where you just feed raw binaries as input to some generic architecture as we can do with images in some tasks.