We _must_ build techniques to continue training existing models, and we have to figure out how to do it in a relatively ubiquitous way. The underlying data tha

We _must_ build techniques to continue training existing models, and we have to figure out how to do it in a relatively ubiquitous way. The underlying data that a model is trained on becomes obsolete relatively quickly - I am constantly running into problems with GPT-4 while trying to solve technical problems, because it's cutoff was 2021 and a ton of code has changed since then, rendering much of the knowledgebase useless. The "paste in the current docs as context" trick only scales so far. This is doubly so for large corporations who will be using "inference" on internal datasets. Training can't be a one-time thing. New documents and state must constantly be added to the weights in order for this technology to be useful in the long run. We need to figure out a way to do this that doesn't constantly make models forget about old training or dramatically overweight recent knowledge.