4 ms·

I believe that so long as weights are fixed at inference time, we'll be at a dead end. Will Titans be sufficiently "neuroplastic" to escape that? Maybe, I'm no

I believe that so long as weights are fixed at inference time, we'll be at a dead end. Will Titans be sufficiently "neuroplastic" to escape that? Maybe, I'm not sure. Ultimately, I think an architecture around "looping" where the model outputs are both some form of "self update" and "optional actionality" such that interacting with the model is more "sampling from a thought space" will be required.

mft_ 2y ago
Very much this. I’ve been wondering why I’ve not seen it much discussed.
- jononor 2y ago
  There are many roadblocks to continual learning still. Most current models and training paradigms are very vulnerable to catastrophic forgetting. And are very sample inefficient. And we/the methods are not so good at separating what is "interesting" (should be learned) vs "not". But this is being researched, for example under the topic of open ended learning, active inference, etc.
  - chriskanan 2y ago
    As a leader in the field of continual learning, I somewhat agree, but I'd say that catastrophic forgetting is largely resolved. The problem is that the continual learning community largely has become insular and is mostly focusing on toy problems that don't matter, where they will even avoid good solutions for nonsensical reasons. For example, reactivation / replay / rehearsal works well for mitigating catastrophic forgetting almost entirely, but a lot of the continual learning community mostly dislikes it because it is very effective. A lot of the work is focusing on toy problems and they refuse to scale up. I wrote this paper with some of my colleagues on this issue, although with such a long author list it isn't as focused as I would have liked in terms of telling the continual learning community to get out of its rut such that they are writing papers that advance AI rather than are just written for other continual learning researchers: https://arxiv.org/abs/2311.11908 https://arxiv.org/abs/2311.11908 The majority are focusing on the wrong paradigms and the wrong questions, which blocks progress towards the kinds of continual learning needed to make progress towards creating models that think in latent space and enabling meta-cognition, which would then give architectures the ability to avoid hallucinations by knowing what they don't know.
    - Nimitz14 2y ago
      Any continual learning papers you're a fan of?
      - chriskanan 2y ago
        Depends on what angle you are interested in. If you are interested in continual learning for something like mitigating model drift such that a model can stay up-to-date where the goal is attain speed ups during training see these works: Compared to other methods for continual learning on ImageNet-1K, SIESTA requires 7x-60x less compute than other methods and achieves the same performance as a model trained in an offline/batch manner. It also works for arbitrary distributions rather than a lot of continual learning methods that only work for specific distributions (and hence don't really match any real-world use case): https://yousuf907.github.io/siestasite/ https://yousuf907.github.io/siestasite/ In this one we focused on mitigating the drop in performance when a system encounters a new distribution. This resulted in a 16x speed up or so: https://yousuf907.github.io/sgmsite/ https://yousuf907.github.io/sgmsite/ In this one, we show how the strategy for creating multi-modal LLMs like LLaVA is identical to a two-task continual learning system and we note that many LLMs once they become multi-modal forget a large amount of the capabilities of the original LLM. We demonstrate that continual learning methods can mitigate that drop in accuracy enabling the multi-modal task to be learned while not impairing uni-modal performance: https://arxiv.org/abs/2410.19925 https://arxiv.org/abs/2410.19925 [We have a couple approaches that are better now that will be out in the next few months] It really depends on what you are interested in. For production AI, the real need is computational efficiency and keeping strong models up-to-date. Not many labs besides mine are focusing on that. Currently, I'm focused on continual learning for creating systems beyond LLMs that incrementally learn meta-cognition and working on continual learning to explain memory consolidation works in mammals and why we have REM phases during sleep, but that's more of a cognitive science contribution so the constraints on the algorithms differ since the goal differs.
        mft_ 2y ago
        > working on continual learning to explain memory consolidation <how> works in mammals and why we have REM phases during sleep That's a nice model: human short-term memory is akin to the context window, and REM sleep consolidating longer-term memories is akin to updating the model itself. How difficult would it be to perform limited focused re-training based on what's been learnt (e.g. new information, new connections, corrections of errors, etc.) within a context window?
- eximius 2y ago
  Self updating requires learning to learn, which I'm not sure we know how to do.
randomNumber7 2y ago
Why, even animals sleep? And if you for example learn an instrument you will notice that a lot of the learning of the muscel memory happens during sleep.
- eximius 2y ago
  I guess you're saying that non-inference time training can be that "sleep period"?
  - randomNumber7 2y ago
    Yes, i could imagine something like a humanoid robot, where the "short term memory" is just a big enough context to keep all input of the day. Then during "sleep" there is training where the information is processed. But I also think that current LLM tech does not lead to agi. You cant train something on pattern matchin and then it becomes magically intelligent (although i could be wrong). Imo an AGI would need to be able to interact with the environment and learn to reflect on its interactions and its abilities within it. I suspect we have the hardware to build s.th. intelligent as a cat or a dog, but not the algorithms.
chriskanan 2y ago
I 100% agree with this and sampling from thought space rather than "thinking" in terms of language. I spent forever writing up an NSF grant proposal on exactly this idea and submitted it last May. I haven't heard back, but it probably won't be funded.

mft_ 2y ago

Very much this. I’ve been wondering why I’ve not seen it much discussed.

jononor 2y ago
There are many roadblocks to continual learning still. Most current models and training paradigms are very vulnerable to catastrophic forgetting. And are very sample inefficient. And we/the methods are not so good at separating what is "interesting" (should be learned) vs "not". But this is being researched, for example under the topic of open ended learning, active inference, etc.
- chriskanan 2y ago
  As a leader in the field of continual learning, I somewhat agree, but I'd say that catastrophic forgetting is largely resolved. The problem is that the continual learning community largely has become insular and is mostly focusing on toy problems that don't matter, where they will even avoid good solutions for nonsensical reasons. For example, reactivation / replay / rehearsal works well for mitigating catastrophic forgetting almost entirely, but a lot of the continual learning community mostly dislikes it because it is very effective. A lot of the work is focusing on toy problems and they refuse to scale up. I wrote this paper with some of my colleagues on this issue, although with such a long author list it isn't as focused as I would have liked in terms of telling the continual learning community to get out of its rut such that they are writing papers that advance AI rather than are just written for other continual learning researchers: https://arxiv.org/abs/2311.11908 https://arxiv.org/abs/2311.11908 The majority are focusing on the wrong paradigms and the wrong questions, which blocks progress towards the kinds of continual learning needed to make progress towards creating models that think in latent space and enabling meta-cognition, which would then give architectures the ability to avoid hallucinations by knowing what they don't know.
  - Nimitz14 2y ago
    Any continual learning papers you're a fan of?
    - chriskanan 2y ago
      Depends on what angle you are interested in. If you are interested in continual learning for something like mitigating model drift such that a model can stay up-to-date where the goal is attain speed ups during training see these works: Compared to other methods for continual learning on ImageNet-1K, SIESTA requires 7x-60x less compute than other methods and achieves the same performance as a model trained in an offline/batch manner. It also works for arbitrary distributions rather than a lot of continual learning methods that only work for specific distributions (and hence don't really match any real-world use case): https://yousuf907.github.io/siestasite/ https://yousuf907.github.io/siestasite/ In this one we focused on mitigating the drop in performance when a system encounters a new distribution. This resulted in a 16x speed up or so: https://yousuf907.github.io/sgmsite/ https://yousuf907.github.io/sgmsite/ In this one, we show how the strategy for creating multi-modal LLMs like LLaVA is identical to a two-task continual learning system and we note that many LLMs once they become multi-modal forget a large amount of the capabilities of the original LLM. We demonstrate that continual learning methods can mitigate that drop in accuracy enabling the multi-modal task to be learned while not impairing uni-modal performance: https://arxiv.org/abs/2410.19925 https://arxiv.org/abs/2410.19925 [We have a couple approaches that are better now that will be out in the next few months] It really depends on what you are interested in. For production AI, the real need is computational efficiency and keeping strong models up-to-date. Not many labs besides mine are focusing on that. Currently, I'm focused on continual learning for creating systems beyond LLMs that incrementally learn meta-cognition and working on continual learning to explain memory consolidation works in mammals and why we have REM phases during sleep, but that's more of a cognitive science contribution so the constraints on the algorithms differ since the goal differs.
      - mft_ 2y ago
        > working on continual learning to explain memory consolidation <how> works in mammals and why we have REM phases during sleep That's a nice model: human short-term memory is akin to the context window, and REM sleep consolidating longer-term memories is akin to updating the model itself. How difficult would it be to perform limited focused re-training based on what's been learnt (e.g. new information, new connections, corrections of errors, etc.) within a context window?
eximius 2y ago
Self updating requires learning to learn, which I'm not sure we know how to do.

randomNumber7 2y ago

Why, even animals sleep? And if you for example learn an instrument you will notice that a lot of the learning of the muscel memory happens during sleep.

eximius 2y ago
I guess you're saying that non-inference time training can be that "sleep period"?
- randomNumber7 2y ago
  Yes, i could imagine something like a humanoid robot, where the "short term memory" is just a big enough context to keep all input of the day. Then during "sleep" there is training where the information is processed. But I also think that current LLM tech does not lead to agi. You cant train something on pattern matchin and then it becomes magically intelligent (although i could be wrong). Imo an AGI would need to be able to interact with the environment and learn to reflect on its interactions and its abilities within it. I suspect we have the hardware to build s.th. intelligent as a cat or a dog, but not the algorithms.

chriskanan 2y ago

I 100% agree with this and sampling from thought space rather than "thinking" in terms of language. I spent forever writing up an NSF grant proposal on exactly this idea and submitted it last May. I haven't heard back, but it probably won't be funded.