4 ms·

I’m not convinced there isn’t a pretty defined ceiling on how clever LLMs can get using current algorithms and hardware.. this isn’t how we think and they all s

I’m not convinced there isn’t a pretty defined ceiling on how clever LLMs can get using current algorithms and hardware.. this isn’t how we think and they all still fall apart on most reasoning tasks. If they haven’t seen lots of examples of something in the training they can’t generalise anything let alone comprehend a book on aerodynamics and design a formula one car. They are more like a parrot than a doctor right now.

marvinkennis 2y ago
A lot of the hype around LLMs and the unfounded promises of them solving complex tasks autonomously seem to stem from a lack of understanding on how these models actually work. The tech is impressive and has its uses, but we shouldn’t pretend like a token predictor can somehow reason and plan.
- smegger001 2y ago
  Sure the current generation, but what happens if we expand its scope, so instead of predicting the next token one at a time, it was allowed greater scope? What happens when you feed it summaries of its work as training data so it has long a short term memory, what happens when you feed several networks together creating a dialogue similar to the theories of a bicameral mind, what if we instead of having the llm halt after every prompt it was put in a loop? Its own output along with outside information used as the prompts for fallowing loop
- textninja 2y ago
  They absolutely can reason and plan; how do you suppose they predict the next token? That they’re not autonomously solving complex tasks is a bit of a straw man though, and with a bit of creativity we can easily imagine them being combined with models and modalities that do provide executive function and autonomy.
  - marvinkennis 2y ago
    This is one of those things I’d love to be wrong on. I think your point on combining them with other models is interesting. Do you think that Markov chains can reason and plan? Dijkstra’s algorithm? Curious where you draw the line
    - textninja 2y ago
      Well, yes, reasoning and planning abilities exist on a spectrum, so it isn’t so much a matter of where to draw the line as a question of degree. As for LLMs, I think their reasoning and planning is some of the most powerful and human-like we’ve seen so far, even if the hidden mechanisms and constraints are different (in some cases, more limited, but in others, vastly superior). Our brains however are highly modular (a “committee of idiots”) so who’s to say a portion, and even a significant one, doesn’t operate on similar principles?
    - Bluestein 2y ago
      For thought: Can DNA reason and plan?
      - textninja 2y ago
        Can a collection of around 1.5 billion interconnected cells that predictably respond to signals in their environment using simple rules? How about 86 billion? 36 trillion? These are ballpark counts of cells in crow’s brain, a human’s brain, and a human body. The question is, is it the cells themselves doing the reasoning and planning, or are they just the machinery this disembodied process happens to be running on? I’d argue intelligence is a distributed phenomenon that our DNA is as much a party to as our brains. Certainly the question of whether humans use DNA to reproduce or DNA uses humans is a matter of perspective.
- IshKebab 2y ago
  I don't know how people can still be making this misguided comment. Predicting tokens well requires reasoning and planning. "Brains are impressive and have their uses, but we shouldn't pretend like a muscle controller can somehow reason and plan."
  - surgical_fire 2y ago
    > Predicting tokens well requires reasoning and planning. Reason and planning regarding token prediction It cannot reason about the context of its output. It only infers the most likely token to follow up.
    - IshKebab 2y ago
      Of course it can reason about the context of its output. I'm honestly not sure what you're trying to say. Can you give me an example of a prompt that requires your definition of "reasoning"?
      - surgical_fire 2y ago
        Have you ever tried to get whatever fancy LLM to write some code for you and had it generate code that was at the same time plausible to look at but complete bullshit? This is what I am talking about. It can reason and plan for what is the likely token based off its training data. It is completely unable to evaluate the logic of the code it was generating. It can not reason on the context (in this case, programming). The same is true for other domains. Law, Medicine, etc. the stricter the field, the less reliable LLMs are, because it cannot reason about the context of what it is writing. That said, I like LLMs, and I think it was an interesting productivity tool, only having been hyped beyond any reasonable expectations. I find it more useful for less strict contexts (for example, creative writing).
  - marvinkennis 2y ago
    I mean obviously predicting tokens in the context of LLM output requires planning. But planning tokens doesn’t generalize. This is evident when you train an LLM on a small data set and ask it about any unseen variation of it. There are many examples of slightly modified popular riddles that are easy to solve by reasoning about them, but LLMs always fall back to the most likely output from their training data.
- rcxdude 2y ago
  There's no reason they can't. In fact LLMs do seem to be the best at artificial general-purpose reasoning of any attempt so far (pretty much every other attempt has failed miserably). That doesn't mean they're particularly good compared to humans, though. I'd characterize them as very high knowledge, low intelligence.
Bluestein 2y ago
(Personally, I am partial to the "alien chef" metaphor for AI: AIs are just reacting the way a chef from Earth would if transported to an alien planet where all the vegetables look the same (and ingredients) but are in reality very different from Earth, and have different effects, including poisonous. They think they are coming up with dishes fit for consumers, but the chef has no understanding of what's going on ...
fnordpiglet 2y ago
The strangeness of this argument is its unnecessary for any one technique to be all in capable because a mixture of techniques is more powerful than one alone. A LLM forming an abductive “plan” and synthesizing results from agent functions that do the deductive reasoning, optimization, math, and other functions LLMs are bad at is remarkably powerful and is incredibly flexible in new semantic contexts. Who cares if LLMs can solve partial differential equations if they can establish a partial differential equation to solve given semantic context and a sufficient training set of semantically grounded PDEs ? That can be delegated to the last 70 years of research on numerical methods. Similarly we don’t expect a recursive decent parser to play chess, but you can absolutely use a parser to make a grammar that exposes a chess playing solver. The combination of the solver and the grammar that enables its use is more powerful than the sum of its parts. LLMs neatly solve an enormous number of hitherto unsolved problems and yield untapped solutions to many more. Generative AI in general composes in powerful ways to unlock even more - semantic analysis of visual information expressible directly in structured language alone is powerful, as is exposing generative robotic motion and audio processing, paired with solvers, optimizers, etc - what’s the residual? Synthetic agency? We’ve had multi level goal based agents for a long time. One that’s reprogrammable by the generative model assembly?

marvinkennis 2y ago

A lot of the hype around LLMs and the unfounded promises of them solving complex tasks autonomously seem to stem from a lack of understanding on how these models actually work. The tech is impressive and has its uses, but we shouldn’t pretend like a token predictor can somehow reason and plan.

smegger001 2y ago
Sure the current generation, but what happens if we expand its scope, so instead of predicting the next token one at a time, it was allowed greater scope? What happens when you feed it summaries of its work as training data so it has long a short term memory, what happens when you feed several networks together creating a dialogue similar to the theories of a bicameral mind, what if we instead of having the llm halt after every prompt it was put in a loop? Its own output along with outside information used as the prompts for fallowing loop
textninja 2y ago
They absolutely can reason and plan; how do you suppose they predict the next token? That they’re not autonomously solving complex tasks is a bit of a straw man though, and with a bit of creativity we can easily imagine them being combined with models and modalities that do provide executive function and autonomy.
- marvinkennis 2y ago
  This is one of those things I’d love to be wrong on. I think your point on combining them with other models is interesting. Do you think that Markov chains can reason and plan? Dijkstra’s algorithm? Curious where you draw the line
  - textninja 2y ago
    Well, yes, reasoning and planning abilities exist on a spectrum, so it isn’t so much a matter of where to draw the line as a question of degree. As for LLMs, I think their reasoning and planning is some of the most powerful and human-like we’ve seen so far, even if the hidden mechanisms and constraints are different (in some cases, more limited, but in others, vastly superior). Our brains however are highly modular (a “committee of idiots”) so who’s to say a portion, and even a significant one, doesn’t operate on similar principles?
  - Bluestein 2y ago
    For thought: Can DNA reason and plan?
    - textninja 2y ago
      Can a collection of around 1.5 billion interconnected cells that predictably respond to signals in their environment using simple rules? How about 86 billion? 36 trillion? These are ballpark counts of cells in crow’s brain, a human’s brain, and a human body. The question is, is it the cells themselves doing the reasoning and planning, or are they just the machinery this disembodied process happens to be running on? I’d argue intelligence is a distributed phenomenon that our DNA is as much a party to as our brains. Certainly the question of whether humans use DNA to reproduce or DNA uses humans is a matter of perspective.
IshKebab 2y ago
I don't know how people can still be making this misguided comment. Predicting tokens well requires reasoning and planning. "Brains are impressive and have their uses, but we shouldn't pretend like a muscle controller can somehow reason and plan."
- surgical_fire 2y ago
  > Predicting tokens well requires reasoning and planning. Reason and planning regarding token prediction It cannot reason about the context of its output. It only infers the most likely token to follow up.
  - IshKebab 2y ago
    Of course it can reason about the context of its output. I'm honestly not sure what you're trying to say. Can you give me an example of a prompt that requires your definition of "reasoning"?
    - surgical_fire 2y ago
      Have you ever tried to get whatever fancy LLM to write some code for you and had it generate code that was at the same time plausible to look at but complete bullshit? This is what I am talking about. It can reason and plan for what is the likely token based off its training data. It is completely unable to evaluate the logic of the code it was generating. It can not reason on the context (in this case, programming). The same is true for other domains. Law, Medicine, etc. the stricter the field, the less reliable LLMs are, because it cannot reason about the context of what it is writing. That said, I like LLMs, and I think it was an interesting productivity tool, only having been hyped beyond any reasonable expectations. I find it more useful for less strict contexts (for example, creative writing).
- marvinkennis 2y ago
  I mean obviously predicting tokens in the context of LLM output requires planning. But planning tokens doesn’t generalize. This is evident when you train an LLM on a small data set and ask it about any unseen variation of it. There are many examples of slightly modified popular riddles that are easy to solve by reasoning about them, but LLMs always fall back to the most likely output from their training data.
rcxdude 2y ago
There's no reason they can't. In fact LLMs do seem to be the best at artificial general-purpose reasoning of any attempt so far (pretty much every other attempt has failed miserably). That doesn't mean they're particularly good compared to humans, though. I'd characterize them as very high knowledge, low intelligence.

Bluestein 2y ago

(Personally, I am partial to the "alien chef" metaphor for AI: AIs are just reacting the way a chef from Earth would if transported to an alien planet where all the vegetables look the same (and ingredients) but are in reality very different from Earth, and have different effects, including poisonous. They think they are coming up with dishes fit for consumers, but the chef has no understanding of what's going on ...

fnordpiglet 2y ago

The strangeness of this argument is its unnecessary for any one technique to be all in capable because a mixture of techniques is more powerful than one alone. A LLM forming an abductive “plan” and synthesizing results from agent functions that do the deductive reasoning, optimization, math, and other functions LLMs are bad at is remarkably powerful and is incredibly flexible in new semantic contexts. Who cares if LLMs can solve partial differential equations if they can establish a partial differential equation to solve given semantic context and a sufficient training set of semantically grounded PDEs ? That can be delegated to the last 70 years of research on numerical methods. Similarly we don’t expect a recursive decent parser to play chess, but you can absolutely use a parser to make a grammar that exposes a chess playing solver. The combination of the solver and the grammar that enables its use is more powerful than the sum of its parts. LLMs neatly solve an enormous number of hitherto unsolved problems and yield untapped solutions to many more. Generative AI in general composes in powerful ways to unlock even more - semantic analysis of visual information expressible directly in structured language alone is powerful, as is exposing generative robotic motion and audio processing, paired with solvers, optimizers, etc - what’s the residual? Synthetic agency? We’ve had multi level goal based agents for a long time. One that’s reprogrammable by the generative model assembly?