4 ms·
Precisely because the exact same codepoints are used for digits and mathematical symbols, there's nothing script-specific about them and their linguistic associ
by yorwba 1y ago
Precisely because the exact same codepoints are used for digits and mathematical symbols, there's nothing script-specific about them and their linguistic association is determined by the training data mixture. A model trained predominantly on text scraped from Chinese websites would learn to associate them more with Mandarin than English in the latent space, since that would be the context where they most often appear.