13 ms·

Is there any evidence that chips degrade in such a way? I've got servers that are 8 years old that ran full bore and they still work fine?

walrus01 4y ago
more concerning is the wear/tear on the fan bearings and the thermal interface between heatsink and board will often be dried out and in poor condition. some gpus are rather hard to get a replacement fan for because it's not like they use an off the shelf modular 60 or 80mm 12vdc fan.
- yardie 4y ago
  The GPU makers aren’t exactly making fans in-house. There is probably a seller on AX with the exact part needed. I’ve had great luck in finding replacement parts for pretty niche hardware.
  - walrus01 4y ago
    if the fans aren't sold by newegg and its top ten largest US domestic competitors as standard sized items - i would consider them as rare weird items, because the size, shape and mounting of the blower is often unique to each card. not going to play games with something I have to order on aliexpress and wait 6 weeks for it to maybe show up from china.
    - KennyBlanken 4y ago
      Most of them are 95mm fans. You can get replacement fans on ebay for almost any relatively common GPU because most of these fans come from the same places and the companies that make the fans for the GPU manufacturers long ago figured out there was a market for replacement units. The higher end card companies in some cases will just ship you a replacement fan if your email to them is pleasant. Or sell it to you for a modest price. I just looked and I can get replacement fans for my specific GPU for less than $20 including shipping.
      - 7speter 4y ago
        You can also take off the Orem fans and shroud and zip tie casefans (noctua or arctic are good brands) onto the heatsink
- jotm 4y ago
  Bah, that's such a minor issue it's not even on the radar. TIM can and should be replaced anyway, fans can be fixed with some automotive grease (not all, sadly). Modding another fan onto it is also an option. The bigger problem is VRM wear, you never know when it will fail. Said TIM drying out affects the VRM components the most. One failed capacitor or resistor and the whole GPU can get fried (seen it happen many times).
  - walrus01 4y ago
    looking on newegg I see lots of factory new 3060, 3070, 3080 selling at pretty regular prices so unless the deal on an ex-mining card was REALLY good (we are not there yet, at price drops in ebay, for when I compare the same NIB newegg price vs used on ebay), I wouldn't buy a used mining card... Maybe if it was like 50% of the new price in a few months from now.
    - jotm 4y ago
      Well, yeah, it'd have to be a great deal.
  - exikyut 4y ago
    :/ Is there a good way to spot this sort of thing from a distance when evaluating stuff off eg eBay?
    - jotm 4y ago
      You could try to guess based on dark spots maybe but that's more like gambling. Boards that smelled "burney" worked fine while others just failed - the cause of failure was heat as in most cases the thermal pads weren't touching the heatsink or the component, were missing or were crumbling. SMD elements can fail either open or closed/shorted, the latter is more likely to cause a worse failure but it's not exclusive. Due to protection circuits elsewhere they can fail without visible marks so it's hard to diagnose. Thermal pads dry out because of heat first, their quality second, with load coming in last imo. So the fact that it's been used for mining alone doesn't mean much. The conditions under which a card was used and the time matter more - if only sellers told you the average temperature, case type, ambient temperatures, whether the cards were undervolted/overclocked and how long they actually ran...
pclmulqdq 4y ago
They do, mostly due to thermal expansion and contraction, fan wear, and thermal paste degradation on the card. Even so, this has a very small effect.
- lostlogin 4y ago
  Wouldn’t constant usage at a set rate reduce these issues?
  - xxs 4y ago
    not for the fans - they are not designed to run at max speed at all times.
magicalhippo 4y ago
> Is there any evidence that chips degrade in such a way? As far as I can see, the answer is yes[1]: The relationship between integrated circuit failure rates and time and temperature is a well established fact. The occurrence of these failures is a function which can be represented by the Arrhenius Model. Well validated and predominantly used for accelerated life testing of integrated circuits, the Arrhenius Model assumes the degradation of a performance parameter is linear with time and that MTBF is a function of temperature stress. However, the dramatic acceleration effect of junction temperature (chip temperature) on failure rate is illustrated in a plot of the above equation for three different activation energies in Figure 2. This graph clearly demonstrates the importance of the relationship of junction temperature to device failure rate. For example, using the 0.99 ev line, a 30° rise in junction temperature, say from 130°C to 160°C, results in a 10 to 1 increase in failure rate. Wikipedia has an overview of the the Arrhenius equation[2]. Now, as you probably know most complex chips like GPUs and CPUs have built-in thermal management which prevents the junction temperature to rise above some limit, which one would assume is set at a point which reasonably guarantees a decent lifetime. However, according to this, chips that have experienced less heat should, on average, live longer. [1]: https://www.ti.com/lit/an/snva509a/snva509a.pdf https://www.ti.com/lit/an/snva509a/snva509a.pdf [2]: https://en.wikipedia.org/wiki/Arrhenius_equation https://en.wikipedia.org/wiki/Arrhenius_equation
- Sakos 4y ago
  I think it's weird how so many people seem to think solid state parts or chips can't fail due to lifetime/heat/use. They don't have obviously moving parts like a fan, but that doesn't mean its materials can't degrade.
  - ip26 4y ago
    They are over-designed enough that failure is quite uncommon for most people.
    - Sakos 4y ago
      I've seen enough failed parts that I know this is misleading if not outright incorrect. It happens. We all have so many electronic devices, many of which we keep for a long time, that the likelihood of your average consumer experiencing a chip failure of some kind is fairly high (over a long enough period of time, probably even guaranteed). Yet for some reason people keep painting it as this statistical impossibility. It's simply not that rare.
digitallyfree 4y ago
The servers are built and rated for 24/7 operation, unlike consumer GPUs. This includes mechanical as well as power components.
nikau 4y ago
From my experience pretty much all failures in modern electronics are due either: * Dried up electrolytic capacitors * Bad solder joints exacerbated by lead free solder. * Failed high current transistors (typically regulation mosfets or triacs in appliances) Failure of low current handling chips is pretty rare.
- esyir 4y ago
  Not sure about the root cause, but there have been numerous reports of mining gpus ram issues, particularly on the radeon VII line. There, after mining, they're often unable to hold even stock memory clocks.
  - nikau 4y ago
    Interesting - I wonder if they have reverted the gpu bios back to a known good OEM image?

walrus01 4y ago

more concerning is the wear/tear on the fan bearings and the thermal interface between heatsink and board will often be dried out and in poor condition. some gpus are rather hard to get a replacement fan for because it's not like they use an off the shelf modular 60 or 80mm 12vdc fan.

yardie 4y ago
The GPU makers aren’t exactly making fans in-house. There is probably a seller on AX with the exact part needed. I’ve had great luck in finding replacement parts for pretty niche hardware.
- walrus01 4y ago
  if the fans aren't sold by newegg and its top ten largest US domestic competitors as standard sized items - i would consider them as rare weird items, because the size, shape and mounting of the blower is often unique to each card. not going to play games with something I have to order on aliexpress and wait 6 weeks for it to maybe show up from china.
  - KennyBlanken 4y ago
    Most of them are 95mm fans. You can get replacement fans on ebay for almost any relatively common GPU because most of these fans come from the same places and the companies that make the fans for the GPU manufacturers long ago figured out there was a market for replacement units. The higher end card companies in some cases will just ship you a replacement fan if your email to them is pleasant. Or sell it to you for a modest price. I just looked and I can get replacement fans for my specific GPU for less than $20 including shipping.
    - 7speter 4y ago
      You can also take off the Orem fans and shroud and zip tie casefans (noctua or arctic are good brands) onto the heatsink
jotm 4y ago
Bah, that's such a minor issue it's not even on the radar. TIM can and should be replaced anyway, fans can be fixed with some automotive grease (not all, sadly). Modding another fan onto it is also an option. The bigger problem is VRM wear, you never know when it will fail. Said TIM drying out affects the VRM components the most. One failed capacitor or resistor and the whole GPU can get fried (seen it happen many times).
- walrus01 4y ago
  looking on newegg I see lots of factory new 3060, 3070, 3080 selling at pretty regular prices so unless the deal on an ex-mining card was REALLY good (we are not there yet, at price drops in ebay, for when I compare the same NIB newegg price vs used on ebay), I wouldn't buy a used mining card... Maybe if it was like 50% of the new price in a few months from now.
  - jotm 4y ago
    Well, yeah, it'd have to be a great deal.
- exikyut 4y ago
  :/ Is there a good way to spot this sort of thing from a distance when evaluating stuff off eg eBay?
  - jotm 4y ago
    You could try to guess based on dark spots maybe but that's more like gambling. Boards that smelled "burney" worked fine while others just failed - the cause of failure was heat as in most cases the thermal pads weren't touching the heatsink or the component, were missing or were crumbling. SMD elements can fail either open or closed/shorted, the latter is more likely to cause a worse failure but it's not exclusive. Due to protection circuits elsewhere they can fail without visible marks so it's hard to diagnose. Thermal pads dry out because of heat first, their quality second, with load coming in last imo. So the fact that it's been used for mining alone doesn't mean much. The conditions under which a card was used and the time matter more - if only sellers told you the average temperature, case type, ambient temperatures, whether the cards were undervolted/overclocked and how long they actually ran...

pclmulqdq 4y ago

They do, mostly due to thermal expansion and contraction, fan wear, and thermal paste degradation on the card. Even so, this has a very small effect.

lostlogin 4y ago
Wouldn’t constant usage at a set rate reduce these issues?
- xxs 4y ago
  not for the fans - they are not designed to run at max speed at all times.

magicalhippo 4y ago

> Is there any evidence that chips degrade in such a way? As far as I can see, the answer is yes[1]: The relationship between integrated circuit failure rates and time and temperature is a well established fact. The occurrence of these failures is a function which can be represented by the Arrhenius Model. Well validated and predominantly used for accelerated life testing of integrated circuits, the Arrhenius Model assumes the degradation of a performance parameter is linear with time and that MTBF is a function of temperature stress. However, the dramatic acceleration effect of junction temperature (chip temperature) on failure rate is illustrated in a plot of the above equation for three different activation energies in Figure 2. This graph clearly demonstrates the importance of the relationship of junction temperature to device failure rate. For example, using the 0.99 ev line, a 30° rise in junction temperature, say from 130°C to 160°C, results in a 10 to 1 increase in failure rate. Wikipedia has an overview of the the Arrhenius equation[2]. Now, as you probably know most complex chips like GPUs and CPUs have built-in thermal management which prevents the junction temperature to rise above some limit, which one would assume is set at a point which reasonably guarantees a decent lifetime. However, according to this, chips that have experienced less heat should, on average, live longer. [1]: https://www.ti.com/lit/an/snva509a/snva509a.pdf https://www.ti.com/lit/an/snva509a/snva509a.pdf [2]: https://en.wikipedia.org/wiki/Arrhenius_equation https://en.wikipedia.org/wiki/Arrhenius_equation

Sakos 4y ago
I think it's weird how so many people seem to think solid state parts or chips can't fail due to lifetime/heat/use. They don't have obviously moving parts like a fan, but that doesn't mean its materials can't degrade.
- ip26 4y ago
  They are over-designed enough that failure is quite uncommon for most people.
  - Sakos 4y ago
    I've seen enough failed parts that I know this is misleading if not outright incorrect. It happens. We all have so many electronic devices, many of which we keep for a long time, that the likelihood of your average consumer experiencing a chip failure of some kind is fairly high (over a long enough period of time, probably even guaranteed). Yet for some reason people keep painting it as this statistical impossibility. It's simply not that rare.

digitallyfree 4y ago

The servers are built and rated for 24/7 operation, unlike consumer GPUs. This includes mechanical as well as power components.

nikau 4y ago

From my experience pretty much all failures in modern electronics are due either: * Dried up electrolytic capacitors * Bad solder joints exacerbated by lead free solder. * Failed high current transistors (typically regulation mosfets or triacs in appliances) Failure of low current handling chips is pretty rare.

esyir 4y ago
Not sure about the root cause, but there have been numerous reports of mining gpus ram issues, particularly on the radeon VII line. There, after mining, they're often unable to hold even stock memory clocks.
- nikau 4y ago
  Interesting - I wonder if they have reverted the gpu bios back to a known good OEM image?