3 ms·

Democratize AI: turn a $95 chip into a 16GB VRAM GPU

13 points by gardenfelder 3y ago

gardenfelder 3y ago
The 4600G is currently selling at price of $95. It includes a 6-core CPU and 7-core GPU. 5600G is also inexpensive — around $130 with better CPU but the same GPU as 4600G.
aimonster2 3y ago
Just get an orange pi 5. For 130ish bucks you get a reasonably fast processor with 16gb(32gb out of stock but soon...) that runs 7b and even 13b models with reasonable speeds for reasonable amounts of context.
kristianp 3y ago
https://archive.is/xtXNB https://archive.is/xtXNB The title is very clickbaity. What's the memory bandwidth of one of these compared to a gpu? Why 16gb of "vram"? Is that the limit of gpu accessible ram? What llm inference performance does it have?
nateb2022 3y ago
[dupe] "AMD Ryzen APU turned into a 16GB VRAM GPU and it can run Stable Diffusion" https://news.ycombinator.com/item?id=37162762 https://news.ycombinator.com/item?id=37162762 (4 months ago, 172pt, 59 comments)
RecycledEle 3y ago
I do wish someone would build a GPU for generative AI applications with some low-power, high-bandwidth RAM (maybe HBM3) and a reasonable SIMD processor. Aim it at being able to run large LLMs from a single PCIe x16 slot at 75 watts. A card that runs an LLM at 1/3 the speed is still worth almost as much to me as a an experimenter. The processor(s) should be clocked down to not burn too much power or emit too much heat in my PC's case. A version of the same card that connects over a very fast USB connection would also be nice. If it could be powered from a wall wart or USB-C PD, that would be awesome.
- kristianp 3y ago
  HBM is very expensive though. I wonder if older generations are still being produced, but significantly cheaper these days? A P100 from 2016 (HBM2) has 732 GB/s memory bandwidth [1], whereas the ddr-4 [2] produces 50GB/s. So the AMD APU would be more than 10x slower at LLM inference. Edit, even old GDDR5 cards like the P40 have bandwidth of 347GB/s. [3] [1] https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.c2888 https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.... [2] https://news.ycombinator.com/item?id=37163639 https://news.ycombinator.com/item?id=37163639 [3] https://www.techpowerup.com/gpu-specs/tesla-p40.c2878 https://www.techpowerup.com/gpu-specs/tesla-p40.c2878

13 points by gardenfelder 3y ago

The 4600G is currently selling at price of $95. It includes a 6-core CPU and 7-core GPU. 5600G is also inexpensive — around $130 with better CPU but the same GPU as 4600G.

aimonster2 3y ago

Just get an orange pi 5. For 130ish bucks you get a reasonably fast processor with 16gb(32gb out of stock but soon...) that runs 7b and even 13b models with reasonable speeds for reasonable amounts of context.

kristianp 3y ago

https://archive.is/xtXNB https://archive.is/xtXNB The title is very clickbaity. What's the memory bandwidth of one of these compared to a gpu? Why 16gb of "vram"? Is that the limit of gpu accessible ram? What llm inference performance does it have?

nateb2022 3y ago

[dupe] "AMD Ryzen APU turned into a 16GB VRAM GPU and it can run Stable Diffusion" https://news.ycombinator.com/item?id=37162762 https://news.ycombinator.com/item?id=37162762 (4 months ago, 172pt, 59 comments)

RecycledEle 3y ago

I do wish someone would build a GPU for generative AI applications with some low-power, high-bandwidth RAM (maybe HBM3) and a reasonable SIMD processor. Aim it at being able to run large LLMs from a single PCIe x16 slot at 75 watts. A card that runs an LLM at 1/3 the speed is still worth almost as much to me as a an experimenter. The processor(s) should be clocked down to not burn too much power or emit too much heat in my PC's case. A version of the same card that connects over a very fast USB connection would also be nice. If it could be powered from a wall wart or USB-C PD, that would be awesome.

kristianp 3y ago
HBM is very expensive though. I wonder if older generations are still being produced, but significantly cheaper these days? A P100 from 2016 (HBM2) has 732 GB/s memory bandwidth [1], whereas the ddr-4 [2] produces 50GB/s. So the AMD APU would be more than 10x slower at LLM inference. Edit, even old GDDR5 cards like the P40 have bandwidth of 347GB/s. [3] [1] https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.c2888 https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.... [2] https://news.ycombinator.com/item?id=37163639 https://news.ycombinator.com/item?id=37163639 [3] https://www.techpowerup.com/gpu-specs/tesla-p40.c2878 https://www.techpowerup.com/gpu-specs/tesla-p40.c2878