3 ms·

This is a gut impression and I don't deny it, but LLMs are Large Language Models, and in my own brain, my Language Model isn't doing large-scale multiplication.

by jerf 1y ago

This is a gut impression and I don't deny it, but LLMs are Large Language Models, and in my own brain, my Language Model isn't doing large-scale multiplication. I have a language-based intuition for the sigle-digit multiplication table and a touch beyond (and based on my observations that's already above average for a human Language Model, at least in my age peer group), but it's not my Language Model doing 283 times 9284. That requires a symbolic manipulation model, and in fact I would observe that my personal neural net, for all the things it is amazingly good at, is in fact quite terrible at that sort of multiplication too. A Commodore PET is by all measures vastly, vastly simpler than my brain, but it blows away my multiplication capabilities. And then the symbolic systems tacked on another, what, 15 orders of magnitude from that "blows away my multiplication capabilities"? Depends on how you count, but something like that. You can sit here and force me to recite ("train me on") multi-digit multiplication problems and their result until the day I die, and my language model is only going to get marginally better. It is in practicing my symbolic manipulation that I'm going to get better and faster. It seems to me that expecting a Language Model to be very good at multiplication is asking for a substantially superhuman level of performance from them, and one that we have little reason to believe will scale anyhow. What we need is symbolic manipulation, better than the approximation they achieve when "reasoning". I find it rather ironic to sit here and use the aforementioned 15 orders of magnitude improvement over the Commodore PET to use that level of symbolic manipulation firepower to laboriously recreate a software system that is as bad as we are at multiplication for what may well be the same fundamental reasons... and then have the audacity to complain about it. My metaphorical dude, you did a couple trillion multiplications just to get to this single bad multiplication output... maybe another approach is called for.

suddenlybananas 1y ago
Language _is_ the symbolic manipulation system par excellence though.
- jerf 1y ago
  There's equivocation in that statement, though, whether you meant there to be or not. There is clearly a difference in how we manipulate English words for normal human activities and the symbolic manipulation with very strict rules we today associate with mathematics and computer science. Human language goes back thousands of years, into the indefinite past we can't track past. Symbolic manipulation is a much, much more recent development, starting only ~2300 years ago around Euclid and not really coming into full development until much later... you can argue about exactly when it is but I'd personally put it as late as the 19th century for it to be recognized in the modern sense. It must be something different if separated by that many centuries. To disprove my point, please generate a list of 5 random 5-digit numbers and demonstrate multiplying them in your head as quickly as you can read them. Since you can't, clearly there is something about that that is hard for you, despite the fact that the act of reading this text, maintaining physical homeostasis while you do it, and all the other things your brain is doing as you do this represents a staggering amount of raw computation that is vastly, vastly in excess of what is nominally needed to achieve that computation.
  - suddenlybananas 1y ago
    Doing multiplication in your head isn't the point though, you can externalise language and use it to do things you can't do in your head by writing it down. Mathematics was born out of very careful reasoning that we do through language, we only use formalisms as they allow us to avoid the massive ambiguities that exist in natural language. Formal symbolic manipulation came out of our already existing abilities of symbolic manipulation through language.
lacy_tinpot 1y ago
A lot of savants that are able to do really cool calculations, or even people that have synesthesia seeing numbers as colors, don't actually do "real" calculations. I think most humans that do math aren't actually literally computing things as some kind of logic machine. We can produce logic, and follow the steps of using that logic, but it doesn't seem to me that our cognition is some kind of logic machine itself.
- daxfohl 1y ago
  True. Generally it seems like you're visualizing things, moving stuff around, seeing vague patterns and trying to make them more clear. IDK how a transformer architecture would fit all of that in its context, or use it productivity once it's there. You can't just keep appending forever, but you also can't delete stuff either, because unlike humans, a deletion is a hard delete; there's no fuzzy remembrance left to rely on, so even deleting bad ideas is dangerous because it'll forget that it was a bad idea and infinite loop. Symbols manipulation doesn't come until the end, after you have a good idea what that part will look like.
- jerf 1y ago
  I'm not sure if you think you're agreeing with me or not, but that is my point. Compared to the nominal account of computational power our brains have, we are staggeringly bad at logical manipulation. We extremely expensively and laboriously simulate them.
daxfohl 1y ago
Hmm, I wonder what happens if you let them manipulate their own context symbolically, maybe something like a stack machine. Perhaps all you need is a "delete" token, or a "replace" flag. That way you don't have context full of irrelevant information. I guess the challenge is, where would the training data come from? Data on the internet is in its final form so "next token" is never a delete. Edit: I guess in essence, that's what reasoning LLMs already do. IIUC the thought blocks are ephemeral, and only the response is maintained for the chat. Maybe there'd be some benefit of doing this recursively? But that's also kind of what subagents are for. So, perhaps nothing new here.
r0x0r007 1y ago
I agree with you, seems like we are trying to make the shoe fit. Not only are we missing the understanding of what is happening inside transformers, but now we are trying to teach them and see how they respond and then interpret it. That seems fine with viruses and animals, but we are talking about a piece of software here. Shouldn't we know what's happening inside? Maybe these kinds of papers can shine more light and give us better understanding though, still it feels backwards to me...Regarding the multiplication itself, shouldn't pure understanding of the meaning of multiplication(it's a summation basically) be enough for 'AI' to call it a day? If AI or human understands that, then the rest is computation part. We already got that covered, so instead of having 'AI' learn it on its own on crazy amount of data and get it right 99% of time, shouldn't we just give it a calculator? Somebody PLEEAASE give this AI a calculator :-)
hodgehog11 1y ago
I think you might be missing some appropriate context. I agree that it is ridiculous to expect a language model to be good at symbolic manipulation; that is best done with tool use. However, there is a significant line of work dedicated to algorithm discovery for mathematical problems using neural networks. Transformers are used here due to their popularity, but also some theoretical analysis to suggest that they are the among the most efficient architecture for learning automata. It's still unclear whether this is truly sound though, which is where this kind of research matters.

by jerf 1y ago

suddenlybananas 1y ago

Language _is_ the symbolic manipulation system par excellence though.

jerf 1y ago
There's equivocation in that statement, though, whether you meant there to be or not. There is clearly a difference in how we manipulate English words for normal human activities and the symbolic manipulation with very strict rules we today associate with mathematics and computer science. Human language goes back thousands of years, into the indefinite past we can't track past. Symbolic manipulation is a much, much more recent development, starting only ~2300 years ago around Euclid and not really coming into full development until much later... you can argue about exactly when it is but I'd personally put it as late as the 19th century for it to be recognized in the modern sense. It must be something different if separated by that many centuries. To disprove my point, please generate a list of 5 random 5-digit numbers and demonstrate multiplying them in your head as quickly as you can read them. Since you can't, clearly there is something about that that is hard for you, despite the fact that the act of reading this text, maintaining physical homeostasis while you do it, and all the other things your brain is doing as you do this represents a staggering amount of raw computation that is vastly, vastly in excess of what is nominally needed to achieve that computation.
- suddenlybananas 1y ago
  Doing multiplication in your head isn't the point though, you can externalise language and use it to do things you can't do in your head by writing it down. Mathematics was born out of very careful reasoning that we do through language, we only use formalisms as they allow us to avoid the massive ambiguities that exist in natural language. Formal symbolic manipulation came out of our already existing abilities of symbolic manipulation through language.

lacy_tinpot 1y ago

A lot of savants that are able to do really cool calculations, or even people that have synesthesia seeing numbers as colors, don't actually do "real" calculations. I think most humans that do math aren't actually literally computing things as some kind of logic machine. We can produce logic, and follow the steps of using that logic, but it doesn't seem to me that our cognition is some kind of logic machine itself.

daxfohl 1y ago
True. Generally it seems like you're visualizing things, moving stuff around, seeing vague patterns and trying to make them more clear. IDK how a transformer architecture would fit all of that in its context, or use it productivity once it's there. You can't just keep appending forever, but you also can't delete stuff either, because unlike humans, a deletion is a hard delete; there's no fuzzy remembrance left to rely on, so even deleting bad ideas is dangerous because it'll forget that it was a bad idea and infinite loop. Symbols manipulation doesn't come until the end, after you have a good idea what that part will look like.
jerf 1y ago
I'm not sure if you think you're agreeing with me or not, but that is my point. Compared to the nominal account of computational power our brains have, we are staggeringly bad at logical manipulation. We extremely expensively and laboriously simulate them.

daxfohl 1y ago

Hmm, I wonder what happens if you let them manipulate their own context symbolically, maybe something like a stack machine. Perhaps all you need is a "delete" token, or a "replace" flag. That way you don't have context full of irrelevant information. I guess the challenge is, where would the training data come from? Data on the internet is in its final form so "next token" is never a delete. Edit: I guess in essence, that's what reasoning LLMs already do. IIUC the thought blocks are ephemeral, and only the response is maintained for the chat. Maybe there'd be some benefit of doing this recursively? But that's also kind of what subagents are for. So, perhaps nothing new here.

r0x0r007 1y ago

I agree with you, seems like we are trying to make the shoe fit. Not only are we missing the understanding of what is happening inside transformers, but now we are trying to teach them and see how they respond and then interpret it. That seems fine with viruses and animals, but we are talking about a piece of software here. Shouldn't we know what's happening inside? Maybe these kinds of papers can shine more light and give us better understanding though, still it feels backwards to me...Regarding the multiplication itself, shouldn't pure understanding of the meaning of multiplication(it's a summation basically) be enough for 'AI' to call it a day? If AI or human understands that, then the rest is computation part. We already got that covered, so instead of having 'AI' learn it on its own on crazy amount of data and get it right 99% of time, shouldn't we just give it a calculator? Somebody PLEEAASE give this AI a calculator :-)

hodgehog11 1y ago

I think you might be missing some appropriate context. I agree that it is ridiculous to expect a language model to be good at symbolic manipulation; that is best done with tool use. However, there is a significant line of work dedicated to algorithm discovery for mathematical problems using neural networks. Transformers are used here due to their popularity, but also some theoretical analysis to suggest that they are the among the most efficient architecture for learning automata. It's still unclear whether this is truly sound though, which is where this kind of research matters.