7 ms·

I do wish someone would build a GPU for generative AI applications with some low-power, high-bandwidth RAM (maybe HBM3) and a reasonable SIMD processor. Aim it

I do wish someone would build a GPU for generative AI applications with some low-power, high-bandwidth RAM (maybe HBM3) and a reasonable SIMD processor. Aim it at being able to run large LLMs from a single PCIe x16 slot at 75 watts. A card that runs an LLM at 1/3 the speed is still worth almost as much to me as a an experimenter. The processor(s) should be clocked down to not burn too much power or emit too much heat in my PC's case. A version of the same card that connects over a very fast USB connection would also be nice. If it could be powered from a wall wart or USB-C PD, that would be awesome.

kristianp 3y ago
HBM is very expensive though. I wonder if older generations are still being produced, but significantly cheaper these days? A P100 from 2016 (HBM2) has 732 GB/s memory bandwidth [1], whereas the ddr-4 [2] produces 50GB/s. So the AMD APU would be more than 10x slower at LLM inference. Edit, even old GDDR5 cards like the P40 have bandwidth of 347GB/s. [3] [1] https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.c2888 https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.... [2] https://news.ycombinator.com/item?id=37163639 https://news.ycombinator.com/item?id=37163639 [3] https://www.techpowerup.com/gpu-specs/tesla-p40.c2878 https://www.techpowerup.com/gpu-specs/tesla-p40.c2878