3 ms·
yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k
by theGnuMe 3y ago
yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k
- kjkjadksj 3y agoYou can set how much of your system ram your gpu gets in windows and linux
- xcv123 3y agoUseless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s
- kjkjadksj 3y agoIf you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.
- xcv123 3y agoM2 Ultra sits between a lower end Intel setup and an H200 for this purpose. I reject this all-or-nothing black and white childish framing of it.