4 ms·
As you can see, that as the "rambling" continues, it increases the number of TOKENs per sentence, and decreases the number of words between TOKENs.
by forlornacorn 3y ago
As you can see, that as the "rambling" continues, it increases the number of TOKENs per sentence, and decreases the number of words between TOKENs.
- robbiep 3y agoApologies and it’s slightly lazy of me to ask, but I was under the impression that a Token was basically 4 bytes/characters of text. This seems to be implying that there’s some differentiation between a token and conjunctions/other sort of in between words?
- forlornacorn 3y agoThat is correct.