3 ms·
Or OpenAI can stop being stupid and adopt LLaMA-like tokenization, which special cases numbers and tokenize them into individual digits.
by sanxiyn 3y ago
Or OpenAI can stop being stupid and adopt LLaMA-like tokenization, which special cases numbers and tokenize them into individual digits.
- jinay 3y agoAre LLaMA-based models better at math because of this digit-based tokenization?
- bugglebeetle 3y agoIsn’t the tokenization tied to the training of the model?
- colobas 3y agoThe model learns an embedding table, where (roughly) each row is used as the model’s internal representation of each token. The numbers in that table are learned. What isn’t learned is the map from token (i.e. combination of characters/byte-pairs) to row-index in the embedding table. That is given by tokenization EDIT: removed redundant bit