There’s no strict definition of a token, it’s just a unit of breaking down text / arbitrary data. You could create a sentence tokenizer, word tokenizer, charact

There’s no strict definition of a token, it’s just a unit of breaking down text / arbitrary data. You could create a sentence tokenizer, word tokenizer, character tokenizer, etc. Afaik this is pretty canonical.