5 ms·

Upstart retrofits an Nvidia GH200 server into a workstation

51 points by jjgreen 3y ago

segmondy 3y ago
Yeah for $51,000. I will happy run a rack of 4090s.
- latchkey 3y ago
  Maybe ok in your house, but if you want to scale out... https://www.nvidia.com/en-us/drivers/geforce-license/ https://www.nvidia.com/en-us/drivers/geforce-license/ No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.
NovemberWhiskey 3y ago
Or: I could just put the OEM version into a rack in another room. Why does it need to be on my desk? It’s not like this is some kind of graphics board that needs to sit a short distance from my monitor.
- deleted 3y ago
  [deleted]
- gessha 3y ago
  A lot of people in Europe don’t have an extra room to put it in. Me neither, signed US metro area dweller.
  - fbdab103 3y ago
    If you were really serious about this, I would think throwing the server into a colocation center would be the way to go. I believe those costs can be as low as a few hundred a month. Security, power, and cooling no longer your problem.
    - tyrfing 3y ago
      Definitely. The amps are the expensive part, but if you have consistently high load, moving from residential rates might mean you're only paying $100-200 for all those other advantages.
  - quickthrower2 3y ago
    50k to drop on a server? You can afford the space or colo.
  - anigbrowl 3y ago
    If you're dropping $50k on a workstation you are either a business or in the upper tier of renters/homeowners. Nobody who can afford this is living in a closet out of necessity.
- almostnormal 3y ago
  > It’s not like this is some kind of graphics board that needs to sit a short distance from my monitor. I wouldn't be surprised if some of the buyers would consider it as gaming rig with maximum bragging rights.
  - miahi 3y ago
    It's not a video card, it does not have video outputs.
buildbot 3y ago
"I started experimenting with Nvidia's RTX 4090s. I bought a bunch of them and put them into a mining rack and just ran some tests. I quickly figured out that is not the way to go," Well, I hope they were at least smart enough to use PCEI4 x16...otherwise that will never, ever work well. 2x 4090 saturate my PCEI4 x16 bus during training workloads. If they were PCEI3 x1 risers... That's 32x lower inter-gpu bandwidth! Also, for 50K per GPU, and all this hacking, buy an older 8x A100 system with water-cooling or something.
- ftufek 3y ago
  Really depends on the model and the software tricks you're using. With DDP and gradient accumulation, you can reduce the bandwidth bottleneck by quite a bit. We've trained with 4090s running at x4 lanes with very small impact. And running at x4 means you can stuff up to 26-28GPUs on a single cpu node (say epyc) and get PCIe latency and get rid of networking hassle.
  - jacquesm 3y ago
    Do you have any pictures and/or documentation of that setup, power draw and performance? It sounds pretty interesting!
    - ftufek 3y ago
      Never got around to writing some public docs. It's essentially bunch of GPUs on custom aluminum extrusion frames sitting in a server rack, connected to romed8-2t motherboard through pcie splitters. Power limited to 240w, negligible performance loss while halving energy usage, uses 3 20a circuits. Performance can range anywhere from 2x4090=1xa100 to 4x4090=1xa100 depending on models, etc. It's great value for the money, and very easy to resell as well.
      - jacquesm 3y ago
        Very nice! 240W? 3 x 20A = 6600W?
moondev 3y ago
> Of course it's bristling with Noctuas – how else do you cool a 1kW desktop? Is this implying that noctuas are good at cooling?
- wmf 3y ago
  Yes?
jrflowers 3y ago
$50k is a steal when you factor in the number of compassionately-worded severance letters you can generate without having to use the Cloud
pella 3y ago
https://gptshop.ai/#info https://gptshop.ai/#info : .. Nvidia GH200 Grace Hopper Superchip-powered supercomputer ... "What is the difference to alternative systems with the same amount of memory? - Compared to 8x Nvidia H100, GH200 costs 5x less, consumes 10x less energy and has roughly the same performance. - Compared to 8x Nvidia A100, GH200 costs 3x less, consumes 5x less energy and has a higher performance. - Compared to 4x AMD Mi300X, GH200 costs 2x less, consumes 4x less energy and has probably roughly the same performance. - Compared to 4x AMD Mi300A (which has only 512 GB memory, more is not possible because the maximum number of scale-up infinity links is 4), GH200 costs significantly less, consumes 3x less energy and has probably a higher performance. - Compared to 8x Nvidia RTX A6000 Ada which has significantly less memory (only 384GB), GH200 costs significantly less, consumes 3x less energy and has a higher performance. - Compared to 8x AMD Radeon PRO W7900 which has significantly less memory (only 384GB), GH200 costs the same, consumes 3x less energy and has a higher performance."
- abstractcontrol 3y ago
  Considering the system only has a single H100, why would it be that performant?
  - buildbot 3y ago
    Yeah this page is full of straight up lies? “ Its performance in every regard is almost unreal (up to 284 times faster than x86).” Like, there are at least 3 things wrong with that statement!
    - pella 3y ago
      benchmark: "NVIDIA GH200 CPU Performance Benchmarks Against AMD EPYC Zen 4 & Intel Xeon Emerald Rapids" * https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmark/5 https://www.phoronix.com/review/nvidia-gh200-gptshop-benchma...
- latchkey 3y ago
  This is a weird comparison written to make things appear only good for GH200. There are a bunch of tradeoffs that aren't considered and some of the comparisons don't make sense since GH200 is a cpu+gpu, so comparing against a gpu only, is weird. There is no such thing as a 4x MI300 chassis, they are all 8x.
DeathArrow 3y ago
€50 000? In 10 years I will buy it for €5000.
Unfrozen0688 3y ago
Someone call Linus Tech Tips right now!!!
solarkraft 3y ago
> ... and due to his preference for keeping work out of the cloud What a German motivation! I too am all for keeping things under your control and this is certainly a very cool exercise ... but I don't quite see why you can't just ... use the server as a server. Then you can connect to it using any portable device. Really, what's the point of "workstations" nowadays, at least for this type of application?

51 points by jjgreen 3y ago

segmondy 3y ago

Yeah for $51,000. I will happy run a rack of 4090s.

latchkey 3y ago
Maybe ok in your house, but if you want to scale out... https://www.nvidia.com/en-us/drivers/geforce-license/ https://www.nvidia.com/en-us/drivers/geforce-license/ No Datacenter Deployment. The SOFTWARE is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.

NovemberWhiskey 3y ago

Or: I could just put the OEM version into a rack in another room. Why does it need to be on my desk? It’s not like this is some kind of graphics board that needs to sit a short distance from my monitor.

deleted 3y ago
[deleted]
gessha 3y ago
A lot of people in Europe don’t have an extra room to put it in. Me neither, signed US metro area dweller.
- fbdab103 3y ago
  If you were really serious about this, I would think throwing the server into a colocation center would be the way to go. I believe those costs can be as low as a few hundred a month. Security, power, and cooling no longer your problem.
  - tyrfing 3y ago
    Definitely. The amps are the expensive part, but if you have consistently high load, moving from residential rates might mean you're only paying $100-200 for all those other advantages.
- quickthrower2 3y ago
  50k to drop on a server? You can afford the space or colo.
- anigbrowl 3y ago
  If you're dropping $50k on a workstation you are either a business or in the upper tier of renters/homeowners. Nobody who can afford this is living in a closet out of necessity.
almostnormal 3y ago
> It’s not like this is some kind of graphics board that needs to sit a short distance from my monitor. I wouldn't be surprised if some of the buyers would consider it as gaming rig with maximum bragging rights.
- miahi 3y ago
  It's not a video card, it does not have video outputs.

buildbot 3y ago

"I started experimenting with Nvidia's RTX 4090s. I bought a bunch of them and put them into a mining rack and just ran some tests. I quickly figured out that is not the way to go," Well, I hope they were at least smart enough to use PCEI4 x16...otherwise that will never, ever work well. 2x 4090 saturate my PCEI4 x16 bus during training workloads. If they were PCEI3 x1 risers... That's 32x lower inter-gpu bandwidth! Also, for 50K per GPU, and all this hacking, buy an older 8x A100 system with water-cooling or something.

ftufek 3y ago
Really depends on the model and the software tricks you're using. With DDP and gradient accumulation, you can reduce the bandwidth bottleneck by quite a bit. We've trained with 4090s running at x4 lanes with very small impact. And running at x4 means you can stuff up to 26-28GPUs on a single cpu node (say epyc) and get PCIe latency and get rid of networking hassle.
- jacquesm 3y ago
  Do you have any pictures and/or documentation of that setup, power draw and performance? It sounds pretty interesting!
  - ftufek 3y ago
    Never got around to writing some public docs. It's essentially bunch of GPUs on custom aluminum extrusion frames sitting in a server rack, connected to romed8-2t motherboard through pcie splitters. Power limited to 240w, negligible performance loss while halving energy usage, uses 3 20a circuits. Performance can range anywhere from 2x4090=1xa100 to 4x4090=1xa100 depending on models, etc. It's great value for the money, and very easy to resell as well.
    - jacquesm 3y ago
      Very nice! 240W? 3 x 20A = 6600W?

moondev 3y ago

> Of course it's bristling with Noctuas – how else do you cool a 1kW desktop? Is this implying that noctuas are good at cooling?

wmf 3y ago
Yes?

jrflowers 3y ago

$50k is a steal when you factor in the number of compassionately-worded severance letters you can generate without having to use the Cloud

pella 3y ago

https://gptshop.ai/#info https://gptshop.ai/#info : .. Nvidia GH200 Grace Hopper Superchip-powered supercomputer ... "What is the difference to alternative systems with the same amount of memory? - Compared to 8x Nvidia H100, GH200 costs 5x less, consumes 10x less energy and has roughly the same performance. - Compared to 8x Nvidia A100, GH200 costs 3x less, consumes 5x less energy and has a higher performance. - Compared to 4x AMD Mi300X, GH200 costs 2x less, consumes 4x less energy and has probably roughly the same performance. - Compared to 4x AMD Mi300A (which has only 512 GB memory, more is not possible because the maximum number of scale-up infinity links is 4), GH200 costs significantly less, consumes 3x less energy and has probably a higher performance. - Compared to 8x Nvidia RTX A6000 Ada which has significantly less memory (only 384GB), GH200 costs significantly less, consumes 3x less energy and has a higher performance. - Compared to 8x AMD Radeon PRO W7900 which has significantly less memory (only 384GB), GH200 costs the same, consumes 3x less energy and has a higher performance."

abstractcontrol 3y ago
Considering the system only has a single H100, why would it be that performant?
- buildbot 3y ago
  Yeah this page is full of straight up lies? “ Its performance in every regard is almost unreal (up to 284 times faster than x86).” Like, there are at least 3 things wrong with that statement!
  - pella 3y ago
    benchmark: "NVIDIA GH200 CPU Performance Benchmarks Against AMD EPYC Zen 4 & Intel Xeon Emerald Rapids" * https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmark/5 https://www.phoronix.com/review/nvidia-gh200-gptshop-benchma...
latchkey 3y ago
This is a weird comparison written to make things appear only good for GH200. There are a bunch of tradeoffs that aren't considered and some of the comparisons don't make sense since GH200 is a cpu+gpu, so comparing against a gpu only, is weird. There is no such thing as a 4x MI300 chassis, they are all 8x.

DeathArrow 3y ago

€50 000? In 10 years I will buy it for €5000.

Unfrozen0688 3y ago

Someone call Linus Tech Tips right now!!!

solarkraft 3y ago

> ... and due to his preference for keeping work out of the cloud What a German motivation! I too am all for keeping things under your control and this is certainly a very cool exercise ... but I don't quite see why you can't just ... use the server as a server. Then you can connect to it using any portable device. Really, what's the point of "workstations" nowadays, at least for this type of application?