2 ms·

> What do you think training to predict when to use different continuations is other than learning? Sure, training = learning, but the problem with LLMs is tha

> What do you think training to predict when to use different continuations is other than learning? Sure, training = learning, but the problem with LLMs is that is where it stops, other than a limited amount of ephemeral in-context learning/extrapolation. With an LLM, learning stops post-training when it is "born" and deployed, while with an animal that's when it starts! The intelligence of an animal is a direct result of it's lifelong learning, whether that's imitation learning from parents and peers (and subsequent experimentation to refine the observed skill), or the never ending process of observation/prediction/surprise/exploration/discovery which is what allows humans to be truly creative - not just behaving in ways that are endless mashups of things they have seen and read about other humans doing (cf training set), but generating truly novel behaviors (such as creating scientific theories) based on their own directed exploration of gaps in mankind's knowledge. Application of AGI to science and new discovery is a large part of why Hassabis defines AGI as human-equivalent intelligence, and understands what is missing, while others like Sam Altman are content to define AGI as "whatever makes us lots of money".

sothatsit 8mo ago
Memory systems built on top of LLMs could provide continual learning. I do not agree that it is some fundamental limitation. Claude Code already writes its own memory files. And people already finetune models. There is clear potential to use the former as a form of short-term memory and the latter for long-term “learning”. The main blockers to this are that models aren’t good enough at managing their own memory, and finetuning is expensive and difficult. But both of these seem like solvable engineering problems.
- HarHarVeryFunny 8mo ago
  Continual learning isn't a "fundamental limitation" or unsolvable problem. Animal brains are an existence proof that it's possible, but it's tough to do, and quite likely SGD is not the way to do it, so any attempt to retrofit continual learning to LLMs as they exist today is going to be a hack... Memory and learning are two different things. Memorization is a small subset of learning. Memorizing declarative knowledge and personal/episodic history (cf. LLM context) are certainly needed, but an animal (or AI intern) also needs to be able to learn procedural skills which need to become baked into the weights that are generating behavior. Fine tuning is also no substitute for incremental learning. You might think of it as addressing somewhat the same goal, but really fine tuning is about specializing a model for a particular use, and if you repeatedly fine tune a model for different specializations (e.g. what I learnt yesterday, vs what I learnt the day before) then you will run into the catastrophic forgetting problem. I agree that incremental learning seems more like an engineering problem rather than a research one, or at least it should succumb to enough brain power and compute put into solving it, but we're now almost 10 years into the LLM revolution (attention paper in 2017) and it hasn't been solved yet - it's not easy.
  - sothatsit 8mo ago
    Fundamentally, I’m more optimistic on how far current approaches can scale. I see no reason why RL could not be used to train models to use memory, and fine-tuning already works, it’s just expensive. The continual learning we get may be a bit hamfisted, and not fit into a neat architecture, but I think we could actually see it work at scale in the next few years. Whereas new techniques like what Yann Lecun have demonstrated still live heavily in the realm of research. Cool, but not useful yet. Fine tuning is also not so limited as you suggest. For one, we don’t need to fine tune the same model over and over, you can just start with a frontier model each time. And two, modern models are much better at generating synthetic data or environments for RL. This could definitely work, but it might require a lot of work in data collection and curation, and the ROI is not clear. But if large companies continue to allocate more and more resources to AI in the next few years, I could see this happening. OpenAI already has a custom model service, and labs have stated they already have custom models built for the military (although how custom those models are is unclear). It doesn’t seem like a huge leap to also fine-tune models over a companies internal codebases and tooling. Especially for large companies like Google, Amazon, or Stripe that employ tens of thousands of software engineers.

sothatsit 8mo ago

Memory systems built on top of LLMs could provide continual learning. I do not agree that it is some fundamental limitation. Claude Code already writes its own memory files. And people already finetune models. There is clear potential to use the former as a form of short-term memory and the latter for long-term “learning”. The main blockers to this are that models aren’t good enough at managing their own memory, and finetuning is expensive and difficult. But both of these seem like solvable engineering problems.

HarHarVeryFunny 8mo ago
Continual learning isn't a "fundamental limitation" or unsolvable problem. Animal brains are an existence proof that it's possible, but it's tough to do, and quite likely SGD is not the way to do it, so any attempt to retrofit continual learning to LLMs as they exist today is going to be a hack... Memory and learning are two different things. Memorization is a small subset of learning. Memorizing declarative knowledge and personal/episodic history (cf. LLM context) are certainly needed, but an animal (or AI intern) also needs to be able to learn procedural skills which need to become baked into the weights that are generating behavior. Fine tuning is also no substitute for incremental learning. You might think of it as addressing somewhat the same goal, but really fine tuning is about specializing a model for a particular use, and if you repeatedly fine tune a model for different specializations (e.g. what I learnt yesterday, vs what I learnt the day before) then you will run into the catastrophic forgetting problem. I agree that incremental learning seems more like an engineering problem rather than a research one, or at least it should succumb to enough brain power and compute put into solving it, but we're now almost 10 years into the LLM revolution (attention paper in 2017) and it hasn't been solved yet - it's not easy.
- sothatsit 8mo ago
  Fundamentally, I’m more optimistic on how far current approaches can scale. I see no reason why RL could not be used to train models to use memory, and fine-tuning already works, it’s just expensive. The continual learning we get may be a bit hamfisted, and not fit into a neat architecture, but I think we could actually see it work at scale in the next few years. Whereas new techniques like what Yann Lecun have demonstrated still live heavily in the realm of research. Cool, but not useful yet. Fine tuning is also not so limited as you suggest. For one, we don’t need to fine tune the same model over and over, you can just start with a frontier model each time. And two, modern models are much better at generating synthetic data or environments for RL. This could definitely work, but it might require a lot of work in data collection and curation, and the ROI is not clear. But if large companies continue to allocate more and more resources to AI in the next few years, I could see this happening. OpenAI already has a custom model service, and labs have stated they already have custom models built for the military (although how custom those models are is unclear). It doesn’t seem like a huge leap to also fine-tune models over a companies internal codebases and tooling. Especially for large companies like Google, Amazon, or Stripe that employ tens of thousands of software engineers.