4 ms·
How does distillation actually work? If I wanted to distill a model and I have enough money and compute, what’s the process?
by andsoitis 1mo ago
How does distillation actually work? If I wanted to distill a model and I have enough money and compute, what’s the process?
- dlcarrier 1mo agoIt mostly involves sorting out Python dependency version conflicts. Also, it takes a lot of compute. It still involves training a model, which for a very small model requires a GPU with several times more ram than the final model size, which will need to run for days to weeks. Usually you're also running the model that is being distilled from, but in this case it would require a huge database of chat logs, instead.