2 ms·
> This entire post and the debate surrounding it, is frankly stupid. What does 95% accuracy even means?? Generally in supervised machine learning a claim of X%
by tzs 10y ago
> This entire post and the debate surrounding it, is frankly stupid. What does 95% accuracy even means??
Generally in supervised machine learning a claim of X% accuracy means that when tested on a large dataset for which the correct result is known and that was not part of the training dataset or validation dataset, it classified X% of that dataset correctly.
Typically you gather a big dataset of labeled data and then split it randomly into training, validation, and test sets. A 50/25/25 split is common. If the learning approach you are using does not need a validation set, then 70/30 training/test is common.
How reliable such an accuracy estimate is depends on how well your dataset matches the characteristics of the datasets people will be using your trained system on. His 95% accuracy report is probably reasonably reliable when his software is used on anonymous posts on the forums where he gathered his datasets. It would probably be less reliable looking at anonymous posts on, say, a bagpipe maker's forum.
- tiredofnoobs 10y agoHuh... of course I know definition of accuracy, and how its calculated. However even in supervised learning, accuracy is only used in very limited cases such as multi class classification. For a whole bunch of problems including the one being discussed its a poor and in some cases a biased metric. E.g. consider a heavily unbalanced problem 99% positive labels. By predicting all instances with majority label its possible to get 99% accuracy. There are several better metrics, False Accept rates, Precision Recall curves etc. Without knowing how the dataset was collected, did the "username" leaked into the dataset, etc. its impossible to evaluate such outlandish claims. The whole moral and ethical debate is non-sequitur, and harms legitimate deep learning research.
- rspeer 10y agoI understand this concern. It's like the old 20 Newsgroups data set for testing classification, where supposedly you're distinguishing the topics of conversation between comp.graphics, sci.electronics, talk.politics.misc, and so on... ...but what the most effective classifiers do is memorize the names and signature blocks of people who posted in each newsgroup.