GPU cores are powerful but you need to feed them data at very high speeds and large models require to pass around a lot of data. This is why Nvidia datacenter p

GPU cores are powerful but you need to feed them data at very high speeds and large models require to pass around a lot of data. This is why Nvidia datacenter products have NVLink and Infiniband so Big Corp can run their massive models. In a recent paper I was reading they were bragging they "only" needed 8 x A100 to train their model. Renting a 4090 24GB costs around $.40/hr. Renting A100 80GB costs around $3.4/hr. For 3x memory and 2x bandwidth. To fit a 13B model in a 4090 you need to quantize it. Imagine a 130B or 200B model. And never mind training, which needs a lot more memory. The divide is crazy. If you are not FAANG you are "GPU-poor".