4 ms·
If they're all equally un-pruned, sounds like they still maintain their linear scale of performance. Just like quantization!
by sodality2 3y ago
If they're all equally un-pruned, sounds like they still maintain their linear scale of performance.
Just like quantization!
- cubefox 3y agoHow does this answer the question?
- danielmarkbruce 3y agoThe question sort of implies you couldn't prune the smaller models and see the same thing. So, the answer given is to consider that in both cases, you sort of only use 30% of the model. Bigger is still bigger. The basic intuition of more parameters = better holds.
- cubefox 3y agoThe question seems to be one of the Chinchilla scaling law. We could train smaller models (than recommended by the law) but with more training tokens, and achieve the same loss. But we would need more compute performance for that. So the question is, perhaps: Why are big models required for compute-efficient training?
- danielmarkbruce 3y agoI don't think so. Pruning a large model and training a smaller model isn't the same thing. It might appear to be the same thing, but it's not.
- cubefox 3y agoDo you expect a model which was overtrained (relative to the Chinchilla law) to be no more affected by pruning than a model of the same size that wasn't overtrained?
- danielmarkbruce 3y agoCan you reformulate this question? It's hard to know what you mean when you say "no more affected". How are you defining "more" ?
- cubefox 3y agoI mean stronger impact on loss or benchmark results.
- danielmarkbruce 3y agoI mean, in some relative (like 10%) or some absolute amount? I think I'd expect the "more trained" model to drop performance by less (as a %, which is hard to define here) but more in absolute sense. Which, is basically impossible to measure but even if it was measurable..I don't feel confident about that prediction, it's speculation.
- bjornsing 3y agoI don’t follow…
- sodality2 3y agoThe article mentions that models have a lot of extra information that is unnecessary. You asked why the large ones still outperform small ones. presumably they all have that inefficiency. But the large ones are still better. 30% of a big number is still bigger than 30% of a small number.