3 ms·
As with most successful applications of machine learning, it's about finessing your approach based on the problem at hand. In our case, we have classes divided
by msalahi 13y ago
As with most successful applications of machine learning, it's about finessing your approach based on the problem at hand. In our case, we have classes divided on the level of "Medicine," "Real Estate," etc. So, we could throw away lots of words that only occurred once or twice in the massive corpus we crawled to build the language model and still have a pretty robust representation of the subject you're trying to represent.
- msalahi 13y agoIn fact, if your training corpus is sufficiently large, you'd be shocked how many words you can eliminate right away for a term frequency of one or two. I went from millions of words in the vocabulary to something like 60k just by ignoring words that happen once or twice in the corpus. Plus, you probably won't learn much about the relationships between words if they only occur a few times in the corpus.
- jlees 13y agoYeah, but consider that some rare words are much stronger indicators of topic than more common ones. Even more so if you look at n-grams. If you use something like wordnet you can get a lot of meaning out of low-frequency words and throw away the meaningless higher-frequency ones that occur in too many categories to be useful.
- adpreese 13y agoSure, there's value in rare words, but I don't think anything that occurs across the corpus fewer than 3 times is going to tell you anything useful. You need a certain amount just to have it be a real signal. What was the least frequent useful word in the data set, msalahi?