3 ms·
Text tokens by their construction are more like the Chinese ideographic characters, so no, there would not be a big difference between tokenized English and Chi
by visarga 3y ago
Text tokens by their construction are more like the Chinese ideographic characters, so no, there would not be a big difference between tokenized English and Chinese. The number of unique tokens is decided upfront - something like 50K - so, much more than basic vocabulary. English tokens can even be 2 word phrases.
Look up Byte-Pair Encoding (BPE) for a fascinating and simple algorithm that explains the process of choosing the tokens. It shows why it doesn't matter how token dense the source language is.