4 ms·
Can someone explain why isn't the word 'the' considered part of the vocabulary in the blog post?
by Jhsto 9y ago
Can someone explain why isn't the word 'the' considered part of the vocabulary in the blog post?
- swyx 9y agousual procedure in NLP: https://en.wikipedia.org/wiki/Stop_words https://en.wikipedia.org/wiki/Stop_words
- neoncontrails 9y agoWord2Vec actually employs a default strategy of pruning the most common 0.0001% of words in the corpus. My understanding is that this permits the RNN layer to train on the unadulterated text rather than an unnatural concentrate of tokens, which is supposedly beneficial. Interestingly there's some controversy whether this benefit extends to punctuation. Most W2V tutorials instruct the programmer to apply tokenizing functions that leave only space-separated word characters, annihilating everything else, but I've actually observed higher accuracy scores from models that tokenize non-word characters individually, i.e. as strings of length 1. I'm not sure exactly why this is the case, but there is a common sense interpretation which might explain. Output from the precursory tokenizer output is often transduced by word2phrase, an optional bigram recognizer. Mikolov strongly recommends using it, but I'm sure it struggles to infer pointwise mutual information from a text that's missing ~70% of its contents.