3 ms·
Ok, I understand. That helped, thanks. And also suddenly the B in BPE makes a lot of sense.
by weinzierl 3y ago
Ok, I understand. That helped, thanks.
And also suddenly the B in BPE makes a lot of sense.
- cubefox 3y agoThe way I think about it: A token can be one to many bytes long, so they can be longer or shorter than a single character.