2 ms·
Very interesting, as I understand it was fine tuned on GSM8k, so we know we can remove 60% of the model and still answer GSM8k questions Bigger question is, do
by rkwasny 3y ago
Very interesting, as I understand it was fine tuned on GSM8k, so we know we can remove 60% of the model and still answer GSM8k questions
Bigger question is, does the sparse model maintain any general knowledge?
- andy99 3y agoIs it compared anywhere with a smaller model fine-tuned on the same thing? Edit: I didn't see any comparisons skimming the paper. More specifically fine tuned smaller models can often be pretty good. I'd want to see how 3B and 1B etc llama models fine-tuned on the same dataset perform. Is sparsity the key here or is it just fewer parameters? The quantization part was interesting - they also quantized activations and have some adaptive quantization that accommodates outliers.