4 ms·

> Clearly we have some sort of goal-based self-correction mechanism. Humans can try things, learn, and iterate. LLMs still can't really do the second thing, yo

> Clearly we have some sort of goal-based self-correction mechanism. Humans can try things, learn, and iterate. LLMs still can't really do the second thing, you can feed back an error message into the prompt but the learning isn't being added to its weights so its knowledge doesn't compound with experience like it does for us. I think there are still a few theoretical breakthroughs needed for LLMs to achieve AGI and one of them is "active learning" like this.

airstrike 1y ago
100% and it seems like we need a whole new architecture to get there, because right now training a model takes so much time. At the risk of making a terrible analogy, right now we're able to "give birth" to these machines after months of training, but once they're born, they can't really learn. Whereas animals learn something new every day, got to sleep, clean up their memories a bit, deleting some, solidifying others, and waking up with an improved understanding of the world.
- bot403 1y ago
  Maybe you're on to something. We need AI lions which will eat the models which don't learn or adapt enough.
  - airstrike 1y ago
    I love the idea of AI lions, but you still need to find a way to allow models to continue "learning" after they're born—which is PhD worthy. Right now we train AI babies, dump them in the wild... and expect them to have all the answers.
psadri 1y ago
You could instruct the LLM to formulate a “lesson” based on the error and add this to the tool instructions for future runs.
- ch4s3 1y ago
  This isn’t practical at scale. You’ll run into too many novel lessons and burn through too many tokens setting up context.
  - psadri 1y ago
    At scale you need to use more tricks. For example, only inject examples if the tool is going to be needed. Or amass lessons, then ask the LLM to summarize them to prune redundant information before it is used in the context.
Vetch 1y ago
Compounding with learn and iterate, humans also build abstractions which significantly shorten the number of steps required. These are more expressive programming languages, compilers and toolchains. We also build engines, libraries, DSLs and invent appropriate data-structures to simplify the landscape or reuse existing work. Besides abstractions, we build tools like better type systems, error testing and borrow checkers to help eliminate certain classes of errors. Finally, after all is said and done, we still have QA teams and major bugs.
cosmic_cheese 1y ago
Additionally, LLMs still don’t truly understand anything, which is why they flounder so badly with e.g. writing code for a programming language or framework that it hasn’t seen a large enough set of training data for. Humans on the other hand do understand and generalize shared knowledge well, which is why we’re much better at handling that type of scenario. More specific to agents, humans can also figure out how to use tools on the fly (even in the absence of documentation) where LLMs need human-built MCPs. This is also a significant limiting factor.
- tkz1312 1y ago
  I’ve found claude to be very helpful when both writing and debugging code written in a language i’m currently building. I just make sure to load the spec into its context first and that seems to be enough for it to get a general understanding.
  - brabel 1y ago
    Everyone criticizing AI for not "understanding" anything... yet, as you found, and many others have also shown before, explain something to them and they bloody well look like they do understand it. I am still in awe at what LLMs can do, TBH. Over the last few months, the main problem with them: of confidently making shit up, seems to be getting much less of a problem... it's still not solved, but if things keep improving I wouldn't be surprised they will have controls that ensure they stop doing that, and when that happens people will be able to trust what they say/write much more... and perhaps that will be a turning point when complaints like in this post will be hard to take seriously.
    - cosmic_cheese 1y ago
      The issue is that their “understanding” following an explanation is quite shallow. They often miss many connections and underlying principles that a human would grasp right away, needing to be spoon-fed these things to fill the gap. That’s not to say they’re not useful in their current state. They are. However, I believe it’s becoming clear that there’s a hard ceiling to how capable LLMs in their current form can become and it’s going to take something radically different to break through.

airstrike 1y ago

100% and it seems like we need a whole new architecture to get there, because right now training a model takes so much time. At the risk of making a terrible analogy, right now we're able to "give birth" to these machines after months of training, but once they're born, they can't really learn. Whereas animals learn something new every day, got to sleep, clean up their memories a bit, deleting some, solidifying others, and waking up with an improved understanding of the world.

bot403 1y ago
Maybe you're on to something. We need AI lions which will eat the models which don't learn or adapt enough.
- airstrike 1y ago
  I love the idea of AI lions, but you still need to find a way to allow models to continue "learning" after they're born—which is PhD worthy. Right now we train AI babies, dump them in the wild... and expect them to have all the answers.

psadri 1y ago

You could instruct the LLM to formulate a “lesson” based on the error and add this to the tool instructions for future runs.

ch4s3 1y ago
This isn’t practical at scale. You’ll run into too many novel lessons and burn through too many tokens setting up context.
- psadri 1y ago
  At scale you need to use more tricks. For example, only inject examples if the tool is going to be needed. Or amass lessons, then ask the LLM to summarize them to prune redundant information before it is used in the context.

Vetch 1y ago

Compounding with learn and iterate, humans also build abstractions which significantly shorten the number of steps required. These are more expressive programming languages, compilers and toolchains. We also build engines, libraries, DSLs and invent appropriate data-structures to simplify the landscape or reuse existing work. Besides abstractions, we build tools like better type systems, error testing and borrow checkers to help eliminate certain classes of errors. Finally, after all is said and done, we still have QA teams and major bugs.

cosmic_cheese 1y ago

Additionally, LLMs still don’t truly understand anything, which is why they flounder so badly with e.g. writing code for a programming language or framework that it hasn’t seen a large enough set of training data for. Humans on the other hand do understand and generalize shared knowledge well, which is why we’re much better at handling that type of scenario. More specific to agents, humans can also figure out how to use tools on the fly (even in the absence of documentation) where LLMs need human-built MCPs. This is also a significant limiting factor.

tkz1312 1y ago
I’ve found claude to be very helpful when both writing and debugging code written in a language i’m currently building. I just make sure to load the spec into its context first and that seems to be enough for it to get a general understanding.
- brabel 1y ago
  Everyone criticizing AI for not "understanding" anything... yet, as you found, and many others have also shown before, explain something to them and they bloody well look like they do understand it. I am still in awe at what LLMs can do, TBH. Over the last few months, the main problem with them: of confidently making shit up, seems to be getting much less of a problem... it's still not solved, but if things keep improving I wouldn't be surprised they will have controls that ensure they stop doing that, and when that happens people will be able to trust what they say/write much more... and perhaps that will be a turning point when complaints like in this post will be hard to take seriously.
  - cosmic_cheese 1y ago
    The issue is that their “understanding” following an explanation is quite shallow. They often miss many connections and underlying principles that a human would grasp right away, needing to be spoon-fed these things to fill the gap. That’s not to say they’re not useful in their current state. They are. However, I believe it’s becoming clear that there’s a hard ceiling to how capable LLMs in their current form can become and it’s going to take something radically different to break through.