3 ms·
Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.
by nverno 4y ago
Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.
- pa7x1 4y agoI just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split. https://imgur.com/a/a3zmkIv https://imgur.com/a/a3zmkIv