5 ms·
I find this article odd with its fixation on computing speed and 8bit. For most current models, you need 40+ GB of RAM to train them. Gradient accumulation doe
by fxtentacle 4y ago
I find this article odd with its fixation on computing speed and 8bit.
For most current models, you need 40+ GB of RAM to train them. Gradient accumulation doesn't work with batch norms so you really need that memory.
That means either dual 3090/4090 or one of the extra expensive A100/H100 options. Their table suggests the 3080 would be a good deal, but it's not. It doesn't have enough RAM for most problems.
If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency.
Also: Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices?
- meragrin_ 4y ago> Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices? So maybe they were including information for the hobbyists/students which do not need or cannot afford the latest and greatest professional cards?
- jszymborski 4y ago> It doesn't have enough RAM for most problems. It might not be as glamorous or make as many headlines, but there is plenty of research that goes on below 40Gb. While I most commonly use A100s for my research, all my models fit on my personal RTX 2080.
- ngcc_hk 4y agoWonder are you trying to walk around the limit or it just happen like this?
- jszymborski 4y agoWe're trying to walk around to limits: I) My research involves biological data (protein-protein interactions) and my datasets are tiny (about 30K high-confidence samples). We have to regularize aggressively and use a pretty tiny network to get something that doesn't over-fit horrendously. II) We want to accommodate many inferences (10^3 to 10^12) inferences on a personal desktop or cheap OVH server in little time so we can serve the model on an online portal.
- ftufek 4y ago> Also: Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices? Out of curiosity, does that also apply for consumer grade GPUs?
- tambre 4y agoI would be surprised if it did. But you probably shouldn't do professional work on GPUs that lack ECC memory.
- nl 4y agoThe lack of ECC memory is almost certainly not a factor. If you can train at FP8 your model will recover from a single flipped bit somewhere.
- chrisMyzel 4y agoyou can get RTX/A6000s but not 3090s or 4090s via inception.
- fxtentacle 4y agoThe client I work with can order 3090 TI and 4090 through their Inception link (in Germany). Apparently, it varies by partner.
- 4y ago
- varunkmohan 4y agoI'm not sure any of this is accurate. 8 bit inference on a 4090 can do 660 Tflops and on an H100 can do 2 Pflops. Not to mention, there is no native support for FP8 (which are significantly better for deep learning) on existing CPUs. The memory on a 4090 can serve extremely large models. Currently, int4 is started to become proven out. With 24GB of memory, you can serve 40 billion parameter models. That coupled with the fact that GPU memory bandwidth is significantly higher than CPU memory bandwidth means that CPUs should rarely ever be cheaper / lower latency than GPUs.
- mota7 4y ago> Gradient accumulation doesn't work with batch norms so you really need that memory. Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated? (precisely because of the need to avoid the memory blowup). Note also that batch normalization can be done in a memory efficient way: It just requires aggregating the batch statistics outside the gradient aggregation.
- fxtentacle 4y agowav2vec2, whisper, HifiGAN, Stable Diffusion, and Imagen all use BatchNorm.
- deleted 4y ago[deleted]
- theLiminator 4y agoAnyone know if the GPUs are relatively affordable through Inception?
- lostmsu 4y ago> Almost everyone using GPUs for work will join NVIDIA's Inception program and get rebates... So why look at retail prices? They need to advertise it better. First time I hear about it. What are the prices like there? GPUs/workstations?
- fxtentacle 4y agoDepends on who you know, but I've seen as low as €799 per new 3090 TI. But you need to waive the right to resell them and there are quotas, for obvious reasons.
- alwayslikethis 4y agoConsumer parts are dirt cheap compared to enterprise ones. Most companies are not able to use them at scale due to CUDA license terms. I don't think there is much of a need for rebates here. For hobbyists, it is somewhat of a steep price for the latest cards, but it's already way down from the height of ETH mining a year back.
- nonbirithm 4y agoThe 4090 Ti is rumored to have 48GB of VRAM, so one can only hope.
- happycube 4y agoThey nerfed the heck out of board memory in the 3000 series (3080 20GB was even made in limited quantity... going to miners in China :( ) so color me a bit skeptical.
- nl 4y ago> For most current models, you need 40+ GB of RAM to train them. Gradient accumulation doesn't work with batch norms so you really need that memory. There's a decision tree chart in the article that addresses this - as it points out there are plenty of models that are much smaller than this. Not everything is a large language model.
- habibur 4y ago> If you can do 8bit inference, don't use a GPU. CPU will be much cheaper and potentially also lower latency. Good advice. Does that mean that I can install like 64 gb ram on a PC and run those models in comparable time?
- fxtentacle 4y agoThat's how cloud speech recognition is usually deployed. OpenAI whisper is faster than realtime on regular desktop CPUs, which I guess is good enough. And for a datacenter, a few $100 AMD CPUs will beat a single $20k NVIDIA A100 at throughput per dollar.