3 ms·
N-grams are unfortunately an overloaded term. It's used in the context of multiple "words" as in this article, but also tuples of characters. The confusion st
by marginalia_nu 3y ago
N-grams are unfortunately an overloaded term.
It's used in the context of multiple "words" as in this article, but also tuples of characters. The confusion stems from the fact that "token" may refer to words as is used in this article (this is common in modern NLP) but older sources often use the word token to refer to graphemes ("letters") and various other breakdowns. This wikipedia article[1] is a good example of such usage.
[1] https://en.wikipedia.org/wiki/Bigram https://en.wikipedia.org/wiki/Bigram
- 3abiton 3y agoDoes it relate to tokenization?
- nostrebored 3y agoThere’s no strict definition of a token, it’s just a unit of breaking down text / arbitrary data. You could create a sentence tokenizer, word tokenizer, character tokenizer, etc. Afaik this is pretty canonical.