5 ms·

Plenty of enterprise server hardware (racks, servers, RAM, disks) does have an active secondhand market after 3-5 years of use, but I think GPUs are too special

by paxys 8mo ago

Plenty of enterprise server hardware (racks, servers, RAM, disks) does have an active secondhand market after 3-5 years of use, but I think GPUs are too specialized for it to be viable. I doubt anyone has the setup to run a H200 in their home rig. I also don't think companies are going to have mandatory replacement cycles for GPU hardware the same way they do for everything else, because: 1. It is an order or magnitude (or more) more expensive. 2. It isn't clear whether Moore's law will apply to the AI GPU space the same way it has for everything else. Unless Nvidia can launch a new chip every 2-3 years with massively improved performance-per-watt at a lower price no one is going to rush to recycle the old one.

tryauuum 8mo ago
you can absolutely run e.g. datacenter-level A100 at home, there are adapters from the SXM to the PCIe socket. Haven't seen people running SXM versions of H100s this way but this could be due to the price factor only
- baby_souffle 8mo ago
  Well technically true, I would wager that the home lab is going to require increasingly distinct and unusual adaptations to retrofit the hardware to home use. New stuff is all liquid cooled by default and that's a paradigm shift for your average home lab. I'm less aware of exactly what's happening on the power side of things but I think some of the architectures are now moving to relatively high voltage DC throughout and then down converting it to low voltage right before it's used. So not exactly just plug-and-play with your average nema15 outlet.
  - tryauuum 8mo ago
    you can still buy air-cooled versions of b300, seems nice to have for a hobbyist
- thefounder 8mo ago
  Well by the time the become obsolete you can run that computing on a Mac with no special cooling so I really doubt they will be of any use. Maybe in some parts of the world where electricity is cheap. If someone wants to really find out perhaps watching the crypto ASICs stories could help.
TacticalCoder 8mo ago
> I doubt anyone has the setup to run a H200 in their home rig. There are PCIe versions of these right? And another comment is saying there are PCI adapters too. It "only" requires 600 to 700W. It's not out of reach for everybody. If the used regular server market is any indication, you can find, after a few years, a lot of enterprise gear at totally discounted prices. CPU costing $4K brand new for $100 after a few years: stuff like that. A friend has got a 42U rack and so do some homelab'ers. People have been running GPU farms mining cryptocurrencies or doing "transcoding" (for money). It's not just CPUs at 1/40th of their brand new price: network gear too. And ECC RAM (before the recent RAM craze). I'm pretty sure that if H200 begin to flood the used market, people shall quickly adapt. > Unless Nvidia can launch a new chip every 2-3 years with massively improved performance-per-watt at a lower price no one is going to rush to recycle the old one. I agree with that. But if they resell old H200s, people are resourceful and shall find a way to run these.
- drivebyhooting 8mo ago
  Where do you find such deals
  - bombcar 8mo ago
    Start on eBay and learn the off-lease companies, and start watching them directly.
- fc417fc802 8mo ago
  Would it even require a particularly high level of resourcefulness? Purchase the GPU along with the mobo that slots it. It's not as though companies typically swap out CPU and GPU while keeping the rest of the box.
  - wmf 8mo ago
    The SXM mobo is huge because it takes 8 GPUs. It requires 10 kW of power etc. You'd be better off with the SXM-PCIe adapters.
epolanski 8mo ago
> Unless Nvidia can launch a new chip every 2-3 years with massively improved performance-per-watt at a lower price no one is going to rush to recycle the old one. That's exactly the point. Performance/watt is increasing so much gen-to-gen that it makes no longer sense to run older hardware. Not my words, Jensen's.
- danpalmer 8mo ago
  Are you saying that the person selling shovels thinks you should buy a new shovel? I guess they must be the expert.
  - alex43578 8mo ago
    Are you saying that there hasn't been massive improvement in performance per watt metrics for datacenter GPUs, which directly affects performance and profitability of said DCs?
    - danpalmer 8mo ago
      No, I'm saying that we should quote numbers and real world examples rather than executives who have a vested interest.
      - alex43578 8mo ago
        You said "I guess they must be the expert", sarcastically, implying NVIDIA's CEO either didn't know or was wrong about the idea of perf/watt improvements. If you have evidence this is wrong, it would be great to present it. Otherwise, most reasonable people will accept that Jensen's statement is accurate, even if he's not a neutral 3rd party.
- kwanbix 8mo ago
  So you are saying that the ceo of the company that builds the chips is saying that it makes sense to change them each generation?
fragmede 8mo ago
K80's are up on eBay.

by paxys 8mo ago

tryauuum 8mo ago

you can absolutely run e.g. datacenter-level A100 at home, there are adapters from the SXM to the PCIe socket. Haven't seen people running SXM versions of H100s this way but this could be due to the price factor only

baby_souffle 8mo ago
Well technically true, I would wager that the home lab is going to require increasingly distinct and unusual adaptations to retrofit the hardware to home use. New stuff is all liquid cooled by default and that's a paradigm shift for your average home lab. I'm less aware of exactly what's happening on the power side of things but I think some of the architectures are now moving to relatively high voltage DC throughout and then down converting it to low voltage right before it's used. So not exactly just plug-and-play with your average nema15 outlet.
- tryauuum 8mo ago
  you can still buy air-cooled versions of b300, seems nice to have for a hobbyist
thefounder 8mo ago
Well by the time the become obsolete you can run that computing on a Mac with no special cooling so I really doubt they will be of any use. Maybe in some parts of the world where electricity is cheap. If someone wants to really find out perhaps watching the crypto ASICs stories could help.

TacticalCoder 8mo ago

> I doubt anyone has the setup to run a H200 in their home rig. There are PCIe versions of these right? And another comment is saying there are PCI adapters too. It "only" requires 600 to 700W. It's not out of reach for everybody. If the used regular server market is any indication, you can find, after a few years, a lot of enterprise gear at totally discounted prices. CPU costing $4K brand new for $100 after a few years: stuff like that. A friend has got a 42U rack and so do some homelab'ers. People have been running GPU farms mining cryptocurrencies or doing "transcoding" (for money). It's not just CPUs at 1/40th of their brand new price: network gear too. And ECC RAM (before the recent RAM craze). I'm pretty sure that if H200 begin to flood the used market, people shall quickly adapt. > Unless Nvidia can launch a new chip every 2-3 years with massively improved performance-per-watt at a lower price no one is going to rush to recycle the old one. I agree with that. But if they resell old H200s, people are resourceful and shall find a way to run these.

drivebyhooting 8mo ago
Where do you find such deals
- bombcar 8mo ago
  Start on eBay and learn the off-lease companies, and start watching them directly.
fc417fc802 8mo ago
Would it even require a particularly high level of resourcefulness? Purchase the GPU along with the mobo that slots it. It's not as though companies typically swap out CPU and GPU while keeping the rest of the box.
- wmf 8mo ago
  The SXM mobo is huge because it takes 8 GPUs. It requires 10 kW of power etc. You'd be better off with the SXM-PCIe adapters.

epolanski 8mo ago

> Unless Nvidia can launch a new chip every 2-3 years with massively improved performance-per-watt at a lower price no one is going to rush to recycle the old one. That's exactly the point. Performance/watt is increasing so much gen-to-gen that it makes no longer sense to run older hardware. Not my words, Jensen's.

danpalmer 8mo ago
Are you saying that the person selling shovels thinks you should buy a new shovel? I guess they must be the expert.
- alex43578 8mo ago
  Are you saying that there hasn't been massive improvement in performance per watt metrics for datacenter GPUs, which directly affects performance and profitability of said DCs?
  - danpalmer 8mo ago
    No, I'm saying that we should quote numbers and real world examples rather than executives who have a vested interest.
    - alex43578 8mo ago
      You said "I guess they must be the expert", sarcastically, implying NVIDIA's CEO either didn't know or was wrong about the idea of perf/watt improvements. If you have evidence this is wrong, it would be great to present it. Otherwise, most reasonable people will accept that Jensen's statement is accurate, even if he's not a neutral 3rd party.
kwanbix 8mo ago
So you are saying that the ceo of the company that builds the chips is saying that it makes sense to change them each generation?

fragmede 8mo ago

K80's are up on eBay.