Yeah I've been giving talks about this: https://www.youtube.com/watch?v=jpWqz85F_4Y https://www.youtube.com/watch?v=jpWqz85F_4Y The biggest problem I see peopl

Yeah I've been giving talks about this: https://www.youtube.com/watch?v=jpWqz85F_4Y https://www.youtube.com/watch?v=jpWqz85F_4Y The biggest problem I see people having is they don't realise that defining a consistent annotation scheme takes a lot of work. If you don't define your problem well, you won't be able to collect consistent annotations, and your model will always perform poorly. You need an annotation scheme that "carves reality at the joints". There will always be boundary cases, but some ways of dividing things up are just bad. For instance, if you want to collect opinions about bands you like on social media, you don't want to have an NER category "BANDS_I_LIKE". That's just dumb: detect bands to handle ambiguities, run the classifier over lots of text, get a frequency list, and then mark the ones you like. Another example. Let's say you were looking for suspicious activity in some dataset like the Panama papers. You might have a bunch of indicators for suspicious activity, like companies that change name a lot, or the same director on companies from very different industries. A lot of people have gotten the message that they should make their models as direct and end-to-end as possible, so they might try to label emails as "INDICATES_SUSPICIOUS_ACTIVITY". This label is unlikely to be the easiest way to do things. For one thing, it's bad to have one category that applies to a bunch of disjunct sets. If you've got a single label for "A or B", you won't have a linear decision boundary. Linearly separable problems are much easier to learn. Another problem here is that the decision you're trying to make rests on tonnes of world knowledge. The business goals would be much better met by learning some simple text annotations like NER, a few simple text categories etc, and using a rule-based approach to stitch things together. Basically, don't just work on having more powerful solutions. Make sure you've tried hard to have easier problems as well --- that part tends to be higher leverage.