The title seems a bit misleading. The paper is about a way to do SFT will less chance of catastrophic forgetting and performance regressions. The idea is that

The title seems a bit misleading. The paper is about a way to do SFT will less chance of catastrophic forgetting and performance regressions. The idea is that SFT on new data that was NOT generated by the model (aka "off policy" data) is likely to cause problems due to the statistical mismatch between the new data and what the model has already learnt. As I understand it, their solution is to statistically align the new data with the old by feeding it to the old model, which will hopefully grok it via in-context learning, then have it regenerate it in its own words such that "off policy" data now becomes "on policy". The model can then be SFT trained on this regenerated data (i.e self-distillation). To me SFT and "continual learning" are two distinct things. Human/animal continual learning is always-on learning that removes the need for, and distinction between, training and inference, and it initiated by prediction failure. It's as much about skill acquisition as it is about knowledge acquisition. Continual learning can happen in any context from trying to do something (or just observing something passively) and being wrong about the outcome of your own actions, or what some external entity does next, to curiosity/boredom driven exploration and play which is more along the spectrum of pure learning with less expectation of outcomes. Continual learning is what, one day, will let the AGI intern pick up new skills on the job by trying to do things and failing/learning/practicing until they get better. This is not the same as sending the intern home with a textbook to read, or a transcript of the conversations you had with it today, and having it take these onboard overnight, which is basically what SFT is designed to do - intermittent addition of new declarative data.