3 ms·
It works because they use really large ngram values, up to 6. So most character-level information is in these subwords.
by stephantul 2y ago
It works because they use really large ngram values, up to 6. So most character-level information is in these subwords.
- p1esk 2y agoLet’s say we want to use 6-grams and build an embedding vector for the word “because”: we add integer vectors for “becaus” and “ecause”, right? For example: [1,2,3,4,5,6] + [2,3,4,5,6,2] = [3,5,7,9,11,8]. Obviously we cannot use this resulting numerical vector to spell the input word. Pretty much all character level info is lost.