3 ms·
Isn't it more that they don't have ready access to the much-more-fundamental concept of decimal numbers? My understanding was that they tokenized them into chu
by Terr_ 2y ago
Isn't it more that they don't have ready access to the much-more-fundamental concept of decimal numbers?
My understanding was that they tokenized them into chunks and tried to learn associations between the chunks, the same as if one was breaking apart English words.
So "2+2=4" isn't being treated that differently from "all's well that ends well." This might lead to a kind of Benny's Rules [0] situation, where sufficient brute-force can make a collection of overfitted non-arithmetic rules appear to work.
[0] https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conception-of.html?m=1 https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conce...
- andrepd 2y agoFascinating article!
- Terr_ 2y agoIt looks like the math-notation formatting didn't survive, for that you might want to see a PDF, ex: https://people.wou.edu/~girodm/library/benny.pdf https://people.wou.edu/~girodm/library/benny.pdf
- isaacfung 2y agoThe current gen llms tokenize numbers digit by digit unlike earlier llms.
- Last5Digits 2y agoThey don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.
- mike_hearn 2y agoNo, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.
- singularity2001 2y agoIf someone else is interested in the Bloomberg tokenizer: https://medium.com/generative-ai-insights-for-business-leaders-and/turning-text-into-data-how-bloomberggpt-tokenizes-financial-language-c17679fb72d0 https://medium.com/generative-ai-insights-for-business-leade...