6 ms·
I wonder if anyone who’s really in the know could summarize where the research is at with getting LLMs to learn “on the job” (through continuous fine tuning or
by libraryofbabel 1y ago
I wonder if anyone who’s really in the know could summarize where the research is at with getting LLMs to learn “on the job” (through continuous fine tuning or whatever) and what the blockers are to this being a useful deployable thing, e.g. having a model+coding agent that can actually learn a codebase over time (cost? model collapse? something else?).
I’m sure this is something the big labs are trying but from the outside as a user of LLMs it feels like people don’t talk about this very much and instead the focus right now is on better training (eg reinforcement learning) with the assumption that anything else not learned during training will be stuffed into the context somehow as needed. But from a naive perspective the lack of learning from experience after training seems like the biggest thing standing between us and AGI.
- ivape 1y agoThe most obvious blocker is compute. This just requires a shit ton more compute.
- libraryofbabel 1y agoThat tracks, but say cost was no object and you had as many H100s as you wanted. Would continuous learning actually work even then?
- IncreasePosts 1y agoMaybe part of the inference outputs could be the updates to make to the network
- johnsmith1840 1y agoIf it was pure compute we'd have simple examples. We can't do this even on the smallest of AI models. There are tons of benchmarks around this you can easily run with 1 gpu. It's compute only in the sense that the only way to do it is retrain a model from scratch at every step. If you solve CL with a CNN you just created AGI.
- Davidzheng 1y agoyeah but training from scratch is a valid solution. And if we can't find easier solutions we should just try to make it work. Compute is the main advantage we have in silica vs biological computers so we might as well push it--like ideally soon we will have one large AI running on datacenter size computer solving really hard problems and it could easily be most of the compute (>95%) is on training step--which is where really AI excels tbh not inference techniques. Like even Alphaproof for example spends most of compute training on solving simpler problems--which btw is one instance of continual training/training at test time which is implemented.
- johnsmith1840 1y agoRetrain from stratch does technically solve it. But it doesn't solve the time aspect. You need to randomize data in order to train to best quality. In doing that the model has no idea t0 was before t1000. If you don't you get model collapse or heavy bias. Some attempts at it but nothing crazy effective.
- zelphirkalt 1y agoHow do you make the mental jump from being able to train a model continuously to an "artificial general intelligence"?
- kadushka 1y agoThe most obvious blocker is catastrophic forgetting.
- solarwindy 1y agoIs that necessarily a blocker? As others in this thread have pointed out, this probably becomes possible only once sufficient compute is available for some form of non-public retraining, at the individual user level. In that case (and hand-waving away just how far off that is), does a model need to retain its generality? Hypothetically (and perhaps more plausibly), a continually learning model that adapts to the context of a particular org / company / codebase / etc., could even be desirable.
- free_bip 1y agoThe most obvious problem is alignment. LLM finetuning is already known to be able to get rid of alignment, so any form of continuous fine tuning would in theory be able to as well.
- notnullorvoid 1y agoWhat kind of alignment are you referring to? Of course more fine-tuning can disrupt earlier fine-tuning, but that's a feature not a bug.
- mnahkies 1y agoI'm no expert, but I'd imagine privacy plays (or should play) a big role in this. I'd expect that compute costs mean any learning would have to be in aggregate rather than specific to the user which would then risk leaking information across sessions very likely. I completely agree that figuring out a safe way to continually train feels like the biggest blocker to AGI
- kcorbitt 1y agoThe real answer is that nobody trusts their automated evals enough to be confident that any given automatically-trained release actually improves performance, even if eval scores go up. So for now everyone batches up updates and vibe-checks them before rolling them out.
- johnsmith1840 1y agoWe have no idea how to do continual learning. Many people here are right, compute, collapse, forgetting whatever. The only "real" way to do this would be: 1. Train a model 2. New data 3. Retrain the model in full + new data 4. Repeat 5. You still have no garuntee on the "time" aspect though. But CL as a field basically has zero answers on how to do this in a true sense. It's crazy hard because the "solutions" are hypocritical in many ways. We need to expand the model's representation space while keeping the previous representation space nearly the same? Basically, you need to modify it without changing it. Most annoying is that even the smallest of natural brains do this easily. I have a long winded theory but basically it boils down to AI likely needs to "sleep" or rest somehow.
- deleted 1y ago[deleted]
- mackenziebowes 1y agoThe cool thing about AI that I'm seeing as an outsider/non-academic, is that it's relatively cheap to clone. Sleeping/resting could be done by a "clone" and benefits could be distributed on a rolling schedule, right?
- johnsmith1840 1y agoOne clone takes a nap while the other works is pretty cool. But the clone couldn't run without sleeping? So that's more of a teammate than a clone. 1 works while the other sleeps and then swap. If this method ever worked our current alignment methods get chucked out the window those would be two completely different AI.
- mackenziebowes 1y agoI can't be certain, I'm not at all an AI engineer or math guy, but I think at the "wake up" point you equalize instances. Like during 'sleep' some list of functions/operations `m` are applied to model weights `n` producing a new model, `n + 1`. Wouldn't you just clone `n + 1`, send it to work, and start a new training run `m + 1` to make `n + 2`?
- karn97 1y ago[dead]