Even if not exactly related to the approach in the article, take a look at adapterhub.ml for training NLP transformer adapters (small modules inserted into larg

Even if not exactly related to the approach in the article, take a look at adapterhub.ml for training NLP transformer adapters (small modules inserted into large pre-trained transformers), that often achieve comparable results to large transformers trained from scratch on many tasks while taking hours to train instead of months.