4 ms·Aren’t Unicode characters generally treated as 2 tokens to avoid a huge vocabulary?by noddybear 4mo agoAren’t Unicode characters generally treated as 2 tokens to avoid a huge vocabulary?