4 ms·
I wonder if we'll get better performance on arithmetic tasks if we let LLMs generate digits backwards, e.g. 12345+11111=<rev>65432</rev> where the rev tags are
by alexvitkov 2y ago
I wonder if we'll get better performance on arithmetic tasks if we let LLMs generate digits backwards, e.g. 12345+11111=<rev>65432</rev> where the rev tags are special tokens.
The reason being that there's less context needed to do it this way, for addition at every step there's only 3 digits that need to be considered and they're already in the token stream.
- astrange 2y agoSince tokens are numbers, I'd like to see the numbers simply tokenized as themselves, or maybe an exponential Golomb code.
- thesz 2y agoMost probably, something like 65432 is just one token.
- alexvitkov 2y agoIt's not, it wouldn't make sense to have 100,000 tokens just for the first 100,000 numbers. There's a playground [1] where you can see how LLMs tokenize a string. 12345678987654321 is tokenized on various models like so: GPT4 123-456-789-876-543-21 GPT3 123-45-678-98-765-43-21 Llama-2, Mistral 1-2-3-4-5-6-7-8-9-8-7-6-5-4-3-2-1 [1] https://huggingface.co/spaces/Xenova/the-tokenizer-playground https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...
- Jerrrrrrry 2y agoLooks like number string parsing may be of enough importance to warrant token look-ahead recursive sub-parsing, then use the most "promising" token-ization; the one that generates the highest yielding probability tree following that number string.
- Vetch 2y agoIt does and the paper mentions some papers that investigate similar strategies in its appendix, but the fundamental precision issues do not go away.