3 ms·

Could you link some of these models? An interesting perspective that asymptote.

PaulHoule 4y ago
I first got involved in text classification in the early 00's and then the best you could do was "bag of word" models that counted the words in a document but didn't take the order of words into account. This works great if you asking a question "Is this paper about astrophysics?" because the vocabulary used in a document is closely linked to the topic. Pretty obviously though if you scramble the words in the document you can't reconstruct the original document, some information is lost, and there are some classification tasks that will reach an upper limit (asymptote) in accuracy because in taking the feature set you lost something. (If the task is "did the defendant commit the crime" the heuristic "Tyrone is a thug" works over bag-of-words, but there is no justice in that.) If that system is able to get the right answer for a case where the word order matters, it just got lucky. You might think "wouldn't it be better to use pairs of words?" but then you run into another problem. You might have a vocabulary of 2,000-20,000 words and get a somewhat useful sample of all of those in a few thousand documents. The number of word pairs is the square of the number of words and you just can't get enough training samples to sample all the possible word pairs. Sentiment analysis was an early area where bag-of-words broke down because I am happy and I am not happy mean very different things. You'd think now that adjectives like "happy" really are special and so is the word "not" and we could make the system somehow realize that "not X" means the opposite of X. You run into an asymptote situation there because there are a huge number of possible negation patterns, for instance you can say I can't say that I am happy and you can't even say "the negation structure has to be within ten words of the adjective" because there is no limit for how complex nested structures can get in language. The first few patterns you add "not X" raise the performance potential of the system a lot but patterns you add after that each make a smaller and smaller contribution to the performance and you again reach an asymptote. Today we have all kinds of embeddings and they are a step forward but they also run into the risk of throwing critical information away, and in a multi-step system you are doomed if an early step does that. I've walked away from some projects where people required high accuracy and they were stuck on using word embeddings that would never attain it. You can think about information loss in embeddings the same way as you do with simpler features except it is a lot more complicated and a lot of people look away instead of confronting the problem.

I first got involved in text classification in the early 00's and then the best you could do was "bag of word" models that counted the words in a document but didn't take the order of words into account. This works great if you asking a question "Is this paper about astrophysics?" because the vocabulary used in a document is closely linked to the topic. Pretty obviously though if you scramble the words in the document you can't reconstruct the original document, some information is lost, and there are some classification tasks that will reach an upper limit (asymptote) in accuracy because in taking the feature set you lost something. (If the task is "did the defendant commit the crime" the heuristic "Tyrone is a thug" works over bag-of-words, but there is no justice in that.) If that system is able to get the right answer for a case where the word order matters, it just got lucky. You might think "wouldn't it be better to use pairs of words?" but then you run into another problem. You might have a vocabulary of 2,000-20,000 words and get a somewhat useful sample of all of those in a few thousand documents. The number of word pairs is the square of the number of words and you just can't get enough training samples to sample all the possible word pairs. Sentiment analysis was an early area where bag-of-words broke down because I am happy and I am not happy mean very different things. You'd think now that adjectives like "happy" really are special and so is the word "not" and we could make the system somehow realize that "not X" means the opposite of X. You run into an asymptote situation there because there are a huge number of possible negation patterns, for instance you can say I can't say that I am happy and you can't even say "the negation structure has to be within ten words of the adjective" because there is no limit for how complex nested structures can get in language. The first few patterns you add "not X" raise the performance potential of the system a lot but patterns you add after that each make a smaller and smaller contribution to the performance and you again reach an asymptote. Today we have all kinds of embeddings and they are a step forward but they also run into the risk of throwing critical information away, and in a multi-step system you are doomed if an early step does that. I've walked away from some projects where people required high accuracy and they were stuck on using word embeddings that would never attain it. You can think about information loss in embeddings the same way as you do with simpler features except it is a lot more complicated and a lot of people look away instead of confronting the problem.