2 ms·
Is it difficult/desirable to squeeze/compress an open-sourced 200B parameter model to fit into 40GB? Are these techniques for specific architectures or can the
by algo_trader 5y ago
Is it difficult/desirable to squeeze/compress an open-sourced 200B parameter model to fit into 40GB?
Are these techniques for specific architectures or can they be made generic ?
- algo_trader 5y agoAh, found some stuff already https://www.tensorflow.org/model_optimization/guide/pruning https://www.tensorflow.org/model_optimization/guide/pruning https://www.tensorflow.org/model_optimization/guide/pruning/pruning_with_keras https://www.tensorflow.org/model_optimization/guide/pruning/...
- ml_hardware 5y agoI think it depends what downstream task you're trying to do... DeepMind tried distilling big language models into smaller ones (think 7B -> 1B) but it didn't work too well... it definitely lost a lot of quality (for general language modeling) relative to the original model. See the paper here, Figure A28: https://kstatic.googleusercontent.com/files/b068c6c0e64d6f933068f7de30ea722359ef87c6c14d3065856b86d44fbdf2dea3ff373ed9eb751514f242d20df9d6a468622fad093f962563545e7d0cdb9dba https://kstatic.googleusercontent.com/files/b068c6c0e64d6f93... But if your downstream task is simple, like sequence classification, then it may be possible to compress the model without losing much quality.