4 ms·

I was aiming more at the centralized-control angle (though I didn't make that very clear), i.e. are open-source models actually viable long-term? If only orgs w

I was aiming more at the centralized-control angle (though I didn't make that very clear), i.e. are open-source models actually viable long-term? If only orgs with absurd amounts of compute can do updates because those imply a full re-training, wouldn't that effectively centralize control over any such model? Is there the option to to an incremental, limited re-training?

ShamelessC 4y ago
Much of modern deep learning is actually premised on the discovery that training on a large, noisy dataset _first_, and then fine tuning (starting training on new data with the same weights) is generally quicker to converge, and also more accurate. This is part of the motivation for “foundation models”. There’s another paradigm called student/teacher models where a randomly initialized model updates it’s weights according to another pretrained model. This could (maybe?) be used to achieve the desired effect of a model that learned in a “clean room”.