I actually authored a patent for something related to this in 2020: https://scholar.google.com/citations?view_op=view_citation&hl=ca&user=l7qMBpkAAAAJ&citation_

I actually authored a patent for something related to this in 2020: https://scholar.google.com/citations?view_op=view_citation&hl=ca&user=l7qMBpkAAAAJ&citation_for_view=l7qMBpkAAAAJ:YsMSGLbcyi4C https://scholar.google.com/citations?view_op=view_citation&h... Not specifically for Transformers but for all kinds of neural networks. It's about training long-lived neural networks where the training spans longer time scales than the neural network architecture remains valid. For example in self-driving cars, it's important to be able to train a model with collected live data over the years so that we can upgrade the model architecture as we go without throwing out the training already done. The patent describes operations where the model architecture can be mutated step by step from old architecture to a new one without forgetting too much of the training which was applied to the old model. This includes but isn't limited to: Increasing/decreasing the width of neuronal layers, adding/removing layers, adding/removing inputs, adding/removing outputs, changing activation functions, changing loss functions. It also describes planning the steps of how to migrate from the old architecture to the new one.