6 ms·
Retraining a large GPT is very expensive. The goal of this paper is to help limit the need for retraining after pruning.
by valine 3y ago
Retraining a large GPT is very expensive. The goal of this paper is to help limit the need for retraining after pruning.
- dougmwne 3y agoExtremely expensive, though as I understand it, the goal is to get maximum performance out of a given model size so that you can actually inference at product scale. A few extra million for training is expensive, but then consider what it costs to run inference for something like Bing for 100 million daily active users. If they can develop new methods to “overtrain” these models they will get more bang out of the smaller parameter model buck.