3 ms·
As the ancestor commenter mentioned: > If you’re interested in ML training Training - at least the one I tried - requires to be run in fp16 mode. So a 7b net
by eurekin 3y ago
As the ancestor commenter mentioned:
> If you’re interested in ML training
Training - at least the one I tried - requires to be run in fp16 mode. So a 7b net needs 14 GB for the model weights alone, plus some extra for the context and the stuff I don't really understand (some gradient values, oh that makes sense now that I've written it)
- BaculumMeumEst 3y agousing cloud hardware for training and consumer cards for inference seems like the common sense thing to do then
- eurekin 3y agoIn the spirit of hackernews, one can also build a rig such as this one: https://nonint.com/2022/05/30/my-deep-learning-rig/ https://nonint.com/2022/05/30/my-deep-learning-rig/ and land a job at OpenAI as a side effect. Of all places, I wasn't expecting such pushback here
- phero_cnstrcts 3y agoI thought it was possible to use two cards and “share” the ram? Then it would still be a good deal. But maybe I’m wrong.
- samspenc 3y agoThis is only supported with the previous generation NVidia 3090, it is apparently possible to combine two 3090s with 24 GB VRAM and 'fuse' them with NVLink to act as a single high-powered GPU with 48 GB VRAM combined. NVidia no longer supports this for the 40-series, I think this is because they want anyone interested in using their GPUs for LLMs to buy the pricier models with more VRAM.
- two_in_one 3y agoTheoretically you can use as many as GPUs you want in parallel. LLMs are easy to split and run in model parallel configuration (for big models which don't fit on one card). or data parallel for performance, when the same model runs different batches on GPUs. PyTorch has full support for both modes, afaik.