4 ms·
Interesting article, but the premise is wrong. You don't need 720GB ram to load the model. The author claims that you need to first load up the model (360GB) an
by leiframming 3y ago
Interesting article, but the premise is wrong. You don't need 720GB ram to load the model. The author claims that you need to first load up the model (360GB) and then the load the weights (another 360 GB) and the replace the first with the latter.
HF allows initializing empty models that do not take up any space using accelerate, especially so that you don't run into that problem.
Still, 180B is a best to load up.
- netdur 3y agolater he said HF's tensor has almost zero copy, so 720GB is not needed using this format
- leiframming 3y agoNo, I think the author gets this mixed up. Zero copy does not play a role here - the problem is how the model is being loaded in the steps that are shown in the article and they have nothing to do with Safe tensors' zero copy ability. Check out the article by Hugging face to see how to do it with only 360GB instead of 720: https://huggingface.co/docs/accelerate/usage_guides/big_modeling https://huggingface.co/docs/accelerate/usage_guides/big_mode...
- kristianp 3y ago> Still, 180B is a best to load up. Best -> Beast