3 ms·
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, wha
by alxlaz 4y ago
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options?
Others who successfully do it have exactly the same secret sauce that you do: they assign it to someone who is well-compensated and competent.
The one time I needed anything remotely like this I just took the old "nobody said programming was gonna be glamorous" adage and ran with it for two weeks. Money-wise, two weeks of programmer time sorting data manually sure beats twelve weeks of programmer time debugging models to cope with inaccurate data, and the results are orders of magnitude better.
Given the widespread understanding of how critical training data is, it's mind-boggling to me that businesses in this field try to outsource it to the lowest-paid external company they can find, thus offering the lowest possible performance incentives and pretty much losing any control they have over quality assurance. Then they proceed to spend humongous amounts of money on clean-up and further refining training data sets, with which they could've hired English Lit majors in the first place, who would've given them a perfectly-classified data set from the very beginning.
- EricMausler 4y agoBut then STEM would have to admit that English Lit Majors didn't waste their money /s