6 ms·
This paper has a good solution: https://arxiv.org/abs/2402.14903 https://arxiv.org/abs/2402.14903 You right to left tokenize in groups of 3, so 1234567 become
by cschmidt 1y ago
This paper has a good solution:
https://arxiv.org/abs/2402.14903 https://arxiv.org/abs/2402.14903
You right to left tokenize in groups of 3, so 1234567 becomes 1 234 567 rather than the default 123 456 7. And if you ensure all 1-3 digits groups are in the vocab, it does much better.
Both https://arxiv.org/abs/2503.13423 https://arxiv.org/abs/2503.13423 and https://arxiv.org/abs/2504.00178 https://arxiv.org/abs/2504.00178 (co-author) both independently noted that you can do this with just by modifying the pre-tokenization regex, without having to explicitly add commas.
- jvanderbot 1y agoOk great! This is precisely how I chunk numbers for comparison. And not to diminish a solid result or the usefulness of it or the baseline tech: its clear that it we keep having to create situation - specific inputs or processes, we're not at AGI with this baseline tech
- chmod775 1y ago> [..] we're not at AGI with this baseline tech DAG architectures fundamentally cannot be AGI and you cannot even use them as a building block for a hypothetical AGI if they're immutable at runtime. Any time I hear the goal being "AGI" in the context of these LLMs, I feel like listening to a bunch of 18th-century aristocrats trying to get to the moon by growing trees. Try to create useful approximations using what you have or look for new approaches, but don't waste time on the impossible. There's no iterative improvements here that will get you to AGI.
- kristjansson 1y ago> "So... what does the thinking?" > "You're not understanding, are you? The brain does the thinking. The meat." > "Thinking meat! You're asking me to believe in thinking meat!" https://www.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html https://www.mit.edu/people/dpolicar/writing/prose/text/think...
- mgraczyk 1y agoThis is meant to be some kind of Chinese room argument? Surely a 1e18 context window model running at 1e6 tokens per second could be AGI.
- lukan 1y ago"Surely a 1e18 context window model running at 1e6 tokens per second could be AGI." And why?
- mgraczyk 1y agoBecause that's quite a bit more information processing than any human brain
- lukan 1y agoI don't think it is quantity that matters. Otherwise supercomputers are smart by definition.
- mgraczyk 1y agoWell no, that's not what anyone is saying. The claim was that it isn't possible in principle for "DAGs" or "immutable architectures" to be intelligent. That statement is confusing some theoretical results that aren't applicable to how LLMs work (output context is mutation). I'm not claiming that compute makes the m intelligent. I'm pointing out that it is certainly possible, and at that level of compute it should be plausible. Feel free to share any theoretical results you think demonstrate the impossibility of "DAG" intelligence and are applicable
- lukan 1y agoI am not saying it is impossible, I am saying it might be possible, but far from plausible with the current approach of LLMs in my experience with them.
- chmod775 1y ago
- AllegedAlec 1y agoThank you. It's maddening how people keep making this fundamental mistake.
- munksbeer 1y agoIt doesn't feel particularly interesting to keep dismissing "these LLMs" as incapable of reaching AGI. It feels more interesting to note that this time, it is different. I've been watching the field since the 90s when I first dabbled in crude neural nets. I am informed there was hype before, but in my time I've never seen progress like we've made in the last five years. If you showed it to people from the 90s, it would be mind blowing. And it keeps improving incrementally, and I do not think that is going to stop. The state of AI today is the worst it will ever be (trivially obvious but still capable of shocking me). What I'm trying to say is that the shocking success of LLMs has become a powerful engine of progress, creating a positive feedback loop that is dramatically increasing investment, attracting top talent, and sharpening the focus of research into the next frontiers of artificial intelligence.
- dTal 1y ago>If you showed it to people from the 90s, it would be mind blowing 90's? It's mind blowing to me now. My daily driver laptop is (internally) a Thinkpad T480, a very middle of the road business class laptop from 2018. It now talks to me. Usually knowledgeably, in a variety of common languages, using software I can download and run for free. It understands human relationships and motivations. It can offer reasonably advice and write simple programs from a description. It notices my tone and tries to adapt its manner. All of this was inconceivable when I bought the laptop - I would have called it very unrealistic sci-fi. I am trying not to forget that.
- nielsole 1y agoIsn't that the opposite of the bitter lesson - adding more cleverness to the architecture?
- cschmidt 1y agoI suppose it is. There is a lot to tokenization - pre-tokenization, how to handle digits, the tokenization training approach - that is about adding cleverness. In the long run, the bitter lesson would be to just get rid of it all and learn from more data. Many people would love to do it. But I think for the case of BLT, digits will still be an issue. There is no way an autoregressive entropy model will be able to split numbers sensibly, since it has no idea how many digits are coming. It seems like it will struggle more with arithmetic. Perhaps you could reverse all the digits in a number, then it has a chance. So 12334 becomes 43321, and it gets to start from the ones digit. This has been suggested as an approach for LLM's.
- infogulch 1y agoLittle endian wins in the end.
- pas 1y ago... why does reversing the all the digits help? could you please explain it? many thanks!
- cschmidt 1y agoMath operations go right to left in the text, while we write them left to right. So if you see the digits 123... in an autoreressive manner, you don't know really anything, since it could be 12345 or 1234567. If you flipped 12345 as 543..., you know the place value of each. You know that the 5 you encounter first is in the ones place, the 4 is the tens place, etc. It gives the LLM a better chance of learning arithmetic.
- pas 1y agoah, okay, thanks! so basically reverse notation has the advantage of keeping magnitude of numbers (digits!) relative to each other constant (or at least anchored to the beginning of the number) doesn't attention help with this? (or, it does help, but not much? or it falls out of autoregressive methods?)
- Y_Y 1y agoWhat do the vector space embeddings for digit strings even look like? Can you do arithmetic on them? If that's even desirable that it seems like you could just skip "embedding" altogether and intern all the numbers along one dimension.