4 ms·

Problem is, those Nvidia servers are useless to anyone who needs servers already racked in a datacenter. They're purposely made purely to run LLMs, and anything

by DiabloD3 11mo ago

Problem is, those Nvidia servers are useless to anyone who needs servers already racked in a datacenter. They're purposely made purely to run LLMs, and anything else would be a waste of time. Nvidia is also fucking over anyone who buy these: datacenters depend on used hardware sales to recoup cost, sometimes getting up to half of what they originally paid for the hardware.... this hardware has no resale value. It's literally garbage the moment it leaves the factory.

eska 11mo ago
2026 is the year of Google Stadia /s
donavanm 11mo ago
> datacenters depend on used hardware sales to recoup cost Who? Ive worked for few big infra companies with millions to billions of DC assets. After depreciation and then some, say 3-5 years, theyre effectively scrap. Its more cost effective to buy new, denser, racks than continue to MRC on the stranded space and power. The reseller is cheaper/easier effectively paid to scrap the parts as e-waste and recover what they can.
- DiabloD3 11mo ago
  You answered your own question. Datacenters hire companies to do this on their behalf, as they don't have the internal know-how to do this. Even the big cloud companies do this. They're on 3-5 year cycles, yet the hardware life has a good 8 years in it. The reseller sifts through what comes out, sees what is still alive, scraps what isn't, and resells the rest. What you don't understand is this isn't a one way relationship. If you're a major cloud company, your hardware probably already is worthless. Meta (and other companies) use a non-standard case and rack "standard" called Open Compute (OCP)... there is no resale market for these, as normal datacenters use normal racks with normal cases. Meta has to pay a company to dispose of these and lose 100% of their investment. But lets say we go to someone else that does use standard hardware, and the rest of the industry buys this stuff up (to maintain existing fleets that don't need upgraded yet, not everybody is on some ridiculous 3-5 year churn), the company you partnered with to deal with your e-waste is likely to either give you a much better rate or even pay you for your hardware (if it is in high demand). These enterprise inference machines have zero resale value. Even the OCP stuff I mentioned above has a small market (theres a few smaller datacenters out there toying with OCP due to it having better density, but aren't willing to buy new to test it out), but there is no market for these inference SBCs. See my sibling comment to this for more information on why the SBCs are uniquely weird.
  - donavanm 11mo ago
    Maybe i read too much in to “depend on used hardware sales.” Ive worked for 2/5 and 3/15 largest us companies doing cloud and infra stuff. Recovered costs from EOL hardware has just never ever mattered. Not even a rounding error on P&L and hardware/dc org has 1000 higher value priorities. Ill admit maybe the offset costs were squirreled away in finance but not visible to the business. Even with zero resale value thats “fine.” Anytime Ive owned capacity planning it’d be more cost effective to pay someone a multiple of rack MRC to get the hardware out and free up the space and whips. The impediment was almost always free hands and coordination functions that were being spent on new adds rather than replacement.
    - DiabloD3 11mo ago
      E-waste disposal is a huge cost, and it might be entirely possible you're not seeing the cost, or you're not aware of what a badly negotiated contract looks like. Also, a lot of the industry runs on incredibly poor margins. The only datacenter space in the world right now printing money is either owned by clouds or owned by the AI bubble (which are sometimes the same companies, or the cloud leasing space to the AI bubble). Mostly, profits are eaten by power deals (this is why Facebook put their biggest important DCs up where the cheapest power in the US is) or property ownership (buying land, building the DC, paying property taxes, maintaining the building, etc, that shit aint cheap), and then you get to buy hardware and hopefully get customers. Amazon, Google, Facebook, et al all cheat their way through every loophole known to man to keep the costs down and the profit high; not a lot of it is from scale, even though they're still trying to chase that to the end, too.
dartharva 11mo ago
> this hardware has no resale value. > It's literally garbage the moment it leaves the factory. Sorry I'm new to this, can you explain how? Why would high-power GPU racks not have good resale value?
- ElectricalUnion 11mo ago
  It's very rare that you'll be at the exact point where it does make sense to use the "not quite high-power GPU racks". * Lowest short term costs favors "just use the Cloud" solutions, that will often have power, HVAC and space constraints that favors new GPUs. * Lowest long term costs bias the cost analysis to OPEX, where old GPUs gets expensive very fast, as they use more power, HVAC and space for less compute.
- DiabloD3 11mo ago
  They're not GPUs in computers... they're effectively GPUs that are computers. They're a single board computer (SBC), with an ARM CPU that is approximately the scale for desktop but not the scale for enterprise, conjoined with 2 extremely large GPUs that are siblings to the desktop version. Each GPU chip is approximately twice the size of the biggest desktop version, and it has two of them, and it roughly twice as dense as anything else you could do using normal enterprise-dense GPU deployments (ie, those rigs that have 4 x16 slots and fit dual-slot GPUs across 8 slots) However, these are not the same chips, they are siblings: the major change is, depending on the generation, they contain 2x or 4x matrix math ALUs and no texture units. Seeing as it is a "small" (for enterprise) CPU, and a GPU that is very poor at non-matrix GPU tasks; and since it doesn't have TUs or media controllers (thats the thing that does DP/HDMI and video decoding/encoding) it can't be some weird niche desktop. Unless you're doing inference at home, this thing is useless. Inference at home, and even small scale enterprise inference, is very very niche. There isn't enough market to give these boards a second chance. However, totally normal used server parts? Totally useful. Cases, fans, power supplies, motherboards, CPUs, RAM, HBA/RAID, NICs? All useful to end users, either smaller companies doing inexpensive onsite, smaller datacenters maintaining an existing fleet and not needing to upgrade yet, or people building home labs.

eska 11mo ago

2026 is the year of Google Stadia /s

donavanm 11mo ago

> datacenters depend on used hardware sales to recoup cost Who? Ive worked for few big infra companies with millions to billions of DC assets. After depreciation and then some, say 3-5 years, theyre effectively scrap. Its more cost effective to buy new, denser, racks than continue to MRC on the stranded space and power. The reseller is cheaper/easier effectively paid to scrap the parts as e-waste and recover what they can.

DiabloD3 11mo ago
You answered your own question. Datacenters hire companies to do this on their behalf, as they don't have the internal know-how to do this. Even the big cloud companies do this. They're on 3-5 year cycles, yet the hardware life has a good 8 years in it. The reseller sifts through what comes out, sees what is still alive, scraps what isn't, and resells the rest. What you don't understand is this isn't a one way relationship. If you're a major cloud company, your hardware probably already is worthless. Meta (and other companies) use a non-standard case and rack "standard" called Open Compute (OCP)... there is no resale market for these, as normal datacenters use normal racks with normal cases. Meta has to pay a company to dispose of these and lose 100% of their investment. But lets say we go to someone else that does use standard hardware, and the rest of the industry buys this stuff up (to maintain existing fleets that don't need upgraded yet, not everybody is on some ridiculous 3-5 year churn), the company you partnered with to deal with your e-waste is likely to either give you a much better rate or even pay you for your hardware (if it is in high demand). These enterprise inference machines have zero resale value. Even the OCP stuff I mentioned above has a small market (theres a few smaller datacenters out there toying with OCP due to it having better density, but aren't willing to buy new to test it out), but there is no market for these inference SBCs. See my sibling comment to this for more information on why the SBCs are uniquely weird.
- donavanm 11mo ago
  Maybe i read too much in to “depend on used hardware sales.” Ive worked for 2/5 and 3/15 largest us companies doing cloud and infra stuff. Recovered costs from EOL hardware has just never ever mattered. Not even a rounding error on P&L and hardware/dc org has 1000 higher value priorities. Ill admit maybe the offset costs were squirreled away in finance but not visible to the business. Even with zero resale value thats “fine.” Anytime Ive owned capacity planning it’d be more cost effective to pay someone a multiple of rack MRC to get the hardware out and free up the space and whips. The impediment was almost always free hands and coordination functions that were being spent on new adds rather than replacement.
  - DiabloD3 11mo ago
    E-waste disposal is a huge cost, and it might be entirely possible you're not seeing the cost, or you're not aware of what a badly negotiated contract looks like. Also, a lot of the industry runs on incredibly poor margins. The only datacenter space in the world right now printing money is either owned by clouds or owned by the AI bubble (which are sometimes the same companies, or the cloud leasing space to the AI bubble). Mostly, profits are eaten by power deals (this is why Facebook put their biggest important DCs up where the cheapest power in the US is) or property ownership (buying land, building the DC, paying property taxes, maintaining the building, etc, that shit aint cheap), and then you get to buy hardware and hopefully get customers. Amazon, Google, Facebook, et al all cheat their way through every loophole known to man to keep the costs down and the profit high; not a lot of it is from scale, even though they're still trying to chase that to the end, too.

dartharva 11mo ago

> this hardware has no resale value. > It's literally garbage the moment it leaves the factory. Sorry I'm new to this, can you explain how? Why would high-power GPU racks not have good resale value?

ElectricalUnion 11mo ago
It's very rare that you'll be at the exact point where it does make sense to use the "not quite high-power GPU racks". * Lowest short term costs favors "just use the Cloud" solutions, that will often have power, HVAC and space constraints that favors new GPUs. * Lowest long term costs bias the cost analysis to OPEX, where old GPUs gets expensive very fast, as they use more power, HVAC and space for less compute.
DiabloD3 11mo ago
They're not GPUs in computers... they're effectively GPUs that are computers. They're a single board computer (SBC), with an ARM CPU that is approximately the scale for desktop but not the scale for enterprise, conjoined with 2 extremely large GPUs that are siblings to the desktop version. Each GPU chip is approximately twice the size of the biggest desktop version, and it has two of them, and it roughly twice as dense as anything else you could do using normal enterprise-dense GPU deployments (ie, those rigs that have 4 x16 slots and fit dual-slot GPUs across 8 slots) However, these are not the same chips, they are siblings: the major change is, depending on the generation, they contain 2x or 4x matrix math ALUs and no texture units. Seeing as it is a "small" (for enterprise) CPU, and a GPU that is very poor at non-matrix GPU tasks; and since it doesn't have TUs or media controllers (thats the thing that does DP/HDMI and video decoding/encoding) it can't be some weird niche desktop. Unless you're doing inference at home, this thing is useless. Inference at home, and even small scale enterprise inference, is very very niche. There isn't enough market to give these boards a second chance. However, totally normal used server parts? Totally useful. Cases, fans, power supplies, motherboards, CPUs, RAM, HBA/RAID, NICs? All useful to end users, either smaller companies doing inexpensive onsite, smaller datacenters maintaining an existing fleet and not needing to upgrade yet, or people building home labs.