3 ms·
has anything been done to like turn common phrases into a single token? like "can you please" maps to 3895 instead of something like "10 245 87 941" Or does i
by tayo42 2y ago
has anything been done to like turn common phrases into a single token?
like "can you please" maps to 3895 instead of something like "10 245 87 941"
Or does it not matter since tokenization is already a kind of compression?
- naveen99 2y agoYou can try cyp but ymmv