4 ms·

"it's nothing to do with solder. it's capacitors"

89 points by robin_reala 2y ago

Rinzler89 2y ago
[flagged]
- meepmorp 2y ago
  These threads are people spontaneously interacting over the internet, not setting out to write articles. You're "eavesdropping," here, and so if the content doesn't seem tailored to you or your better understanding of, it's because it wasn't actually intended for your consumption.
  - 1317 2y ago
    then why was it posted here, in a public forum for the discussion of (technical) articles?
    - meepmorp 2y ago
      Because other people here might find it interesting. The world isn't geared towards you.
jetbalsa 2y ago
That is really interesting! I always though those things where solid state and wouldn't age over time! It makes a ton more sense that there is a somewhat of a reset that is randomly happening when people are fixing stuff nearby and thinking their main target was the thing that fixed it.
- fuzzfactor 2y ago
  Keep in mind that aluminum electrolytic capacitors are not solid-state devices themselves, actually liquid-state in a somewhat similar way to lead-acid batteries.
- falcolas 2y ago
  > I always though those things where solid state and wouldn't age over time! FWIW, thanks to a number of issues, even solid state devices will fail over time. https://c3.ndc.nasa.gov/dashlink/static/media/other/ObservedFailures37.html https://c3.ndc.nasa.gov/dashlink/static/media/other/Observed... (This entire presentation is worth the read, by the way).
- lxgr 2y ago
  Many solid state things age: Plastic and rubber can get weird/gross to the touch over the course of a few years or change color (due to additive breakdown/outgassing, UV radiation etc.). Flash storage cells can lose their stored information (over time) or their capacity to retain it (due to wear, i.e. erase cycles that are each slightly destructive to the materials) etc – and we even use them in things called "solid state drives" :)
- kjs3 2y ago
  things where solid state and wouldn't age over time Common misconception. Most solid state tech will fail over time, with wildly different timescales depending on the tech. For example, semiconductor technology is subject to electromigration, thermal effects and other material failures. As design geometries get smaller, these effects get worse.
nubinetwork 2y ago
It's not the capacitors in 360s that failed... it's something in the GPU, whether it's the interconnect, the die, or bad underfill, people have been trying to figure it out for years... but the failure analysis tools are expensive, and Microsoft will never come clean on why they failed...
- jetbalsa 2y ago
  He is making a good bit of sense, if it was the tiny 0402 SMD Caps near the GPU going south that would explain a good bit of what is going on
  - NikkiA 2y ago
    Ceramic capacitors rarely die, when people talk about recapping electronics it's the electrolytic capacitors that they're talking about; which dry out.
    - robin_reala 2y ago
      Not dying, but falling out of tolerance specification.
      - eqvinox 2y ago
        That's equally rare for MLCCs, in particular for mass decoupling caps. It's really the Al-electrolytics and Tantalums where this happens.
- phendrenad2 2y ago
  The prevailing theory back when I was in the scene was that the GPU was warping the PCB itself, causing solder balls to lose connection. Heating in an oven softens the entire board, allowing it to un-torque.
  - loa_in_ 2y ago
    My theory was that high operating temperature caused the balling to get partially softened and develop crystalline defects when cooling. I base it on personal experience that they never failed while running, but at least three people I know experienced these failures after powering down PCs for few days. All four GPUs were fixed by a friend of mine by... cooking them in the oven briefly
hellojebus 2y ago
I used to be local teen that would fix everyones Playstation 3's, when they got the red light of death. I used to do the heat gun trick to get them back up, but decided to also try an oven at one point (it worked). I wonder if this was the same phenomenon...
- nubinetwork 2y ago
  My understanding is that the ps3 and 360 had similar (but not entirely identical) issues, but the common "repair" on the ps3 was to change some unique (NEC tokin) capacitors around the CPU/GPU... only problem was that, like the hotbox "repair" it would fail again within a month if it worked at all.
- sebazzz 2y ago
  I fixed my NVIDIA video card by putting it in the oven. I wonder if I fixed the capacitors instead of the solder.
  - spacebacon 2y ago
    I’ve exhausted options on my 4090. I sense a need for a “can you bake it” YT channel.
jauntywundrkind 2y ago
Long post, so sharing the key part, > if you heat the ceramic [capacitor] above its Curie temperature, all of the titanium atoms go back into a regular cubic lattice. if you cool it rapidly (just heat sinking away to the board is fine) it resets the aging process. and you can do this infinity times. suddenly the capacitance is back up to where it was before it even left the factory. > the Curie temperature is pretty low. like, 120°C ish. https://chaos.social/@gsuberland/113084484599859593 https://chaos.social/@gsuberland/113084484599859593 Worth reading it all!
renewiltord 2y ago
The nVidia G84 series back in the late 2000s had this problem and that was definitely the BGA solder. They reissued the GPUs after fixing.
fuzzfactor 2y ago
As a message to future visitors who may think "audio" capacitors; For those who are using ceramic coupling caps in their audio circuits, this can help explain some of the differences that can be detected between different types of caps. Especially in vintage-style vacuum tube assemblies where I would expect sometimes the internal temperature does rise above 120 C, and this would depend on ambient temperature too.
- mystified5016 2y ago
  Fun fact, ceramic capacitors (MLCC in particular) have microphonic effects. Vibration and mechanical strain in a capacitor generate a very slight voltage.
paulgerhardt 2y ago
The author is conflating a different contemporaneous issue of the PS3 (YLOD) with the XBox 360. It wasn't "capacitors". Further, I would argue it wasn't monocausal but a big finger can be pointed at thermal cycling fatigue. In broad strokes, a number of issues caused the Red Ring of Death ("general hardware failure") but the most dominant/infamous one was eventually attributed to a cross-domain electro-mechanical issue causing connections to break in the GPU interposer from thermal cycling in the first generation xbox gpu chips (made at a 90nm process node) which were rushed to market and fixed with second generation chips (made at 65nm process node)[1]. Initial attempts to individually address the solder, heat delta (by reducing the max temperture), and mechanical design of these chips did not fix the issue. Ultimately all three had to be sorted. The "towel trick" worked by making the resin in the PCB temporarily more pliant - allowing the x-clamp to reseat the gpu and temporarily re-bridging these broken connections on the GPU and to a lesser extant the CPU. Similar idea when swapping out a stronger "x-clamp" - aftermarket or with washers. Old leadership was given two shots to fix this initially and failed with Zephyr[early 2007] and Falcon[late 2007] generations which were best efforts given the cruel reality that GPU architecture turnaround time of around 2 to 3 years. Ultimately the fix had to come from three sources: ATI (new chips), replacing Peter Moore with Don Mattrick (new leadership and comms), and new hardware revision (Jasper[2008]). Earlier attempts to "replace the lead-free solder with a different formulation" in 2007 were ineffective in addressing the problem. Fundamentally it was a calculated gamble - Moore got a huge (albeit short-sighted) win for himself and Microsoft doubled their game console market share by pushing their vendors and themselves to compromise their release engineering schedule. It's debatable if they would have captured more share by running a more diligent process - probably no - given the issues were initially latent and didn't manifest for a few years, but certainly they would have lost less market share later on in 2008 and 2009. They couldn't, for instance, have known the PS3 would be so expensive at launch nor that the Wii would be such a sleeper hit in an untapped market segment. [1] https://www.youtube.com/watch?v=z2d6IMBS8oY&t=1130s https://www.youtube.com/watch?v=z2d6IMBS8oY&t=1130s
- deleted 2y ago
  [deleted]
- krackers 2y ago
chomp 2y ago
I don’t think this post is accurate, there were multiple problems with the 360 design, it wasn’t capacitors. The towel trick warped the board and sometimes it worked and most of the time you wound up with a warped board and a system that continues to RROD. I fixed mine this way and still have it at my parent’s house, I’ll take a picture when I can, it’s very obvious what happens with the towel trick.
fuzzfactor 2y ago
With solder in the title this might be a good time to remember how much less of a conductor solder is compared to copper. This also may not apply very much to gaming consoles in particular. If the wire carrying high current has copper that is too thin gage, it will heat to some extent since its resistance is not quite zero, and at any one current the more resistance then the more heat will need to be dissipated. If there is a point where it is not pure copper from end-to-end but has a solder joint in between, the solder will be the point of highest resistance so it will autoheat to a higher temperature than the copper does. Plus when a solder joint cools properly it solidifies as a eutectic alloy for the most part rather than a crystalline solid. A "cold" solder joint can have more of a crystalline nature, plus can have lots of metal oxides included if it was allowed to remain molten for too long or at much higher temperature than necessary considering its melting point. IOW a cold solder joint can be even less of a conductor than normal, so much less it can even sometimes act as a bit of a ~semi~conductor in exactly the spot you would least want that.
- mystified5016 2y ago
  Resistance is a function of cross-sectional area of a conductor. Current capacity is a function of resistance, length, and heat dissipation (largely surface area). The cross-sectional area of a copper trace is on the order of ten micrometers square. The cross-sectional area of a solder joint is usually on the order of one or two square millimeters, if not more. For a BGA ball, you can approximate it as a cylinder with radius equal to the ball radius. Again, this is on the scale of millimeters. No matter what, your solder joint will have an order of magnitude larger area than the trace feeding it. Solder joints generally have far lower resistance than a copper trace due to this fact. Besides that, many (most?) lead-free solder alloys use silver, which is famously quite a remarkable conductor. Cold joints and the other problems you mention are basically irrelevant electrically. They're mechanical problems that are likely to cause a break in the circuit. Any added resistance from inclusions or crystal structure are completely irrelevant unless you're doing something extremely specialized like high sensitivity metrology or crazy RF stuff.
somat 2y ago
On the other side of the balance are the "it has nothing to do with the capaciters it is the solder" crowd. https://www.youtube.com/watch?v=I0UMG3iVYZI https://www.youtube.com/watch?v=I0UMG3iVYZI (ripfelix) It is a long watch, but it goes into a fairly deep dive into the whole affair. The author came to the conclusion that while a small ratio of the problem was the caps or the bga, There was a new process in play where chiplets were bonded to a base(bumps) which was then bonded to the motherboard(bga). And all processing units made with this first generations bonding process had a higher then expected failure rate, That is, the ps3, the xbox360 and nvidea graphics cards of this generation are all failing for the same reason. Having said that, That post is an amazing deep dive into the chemistry of capacitors.

89 points by robin_reala 2y ago

Rinzler89 2y ago

[flagged]

meepmorp 2y ago
These threads are people spontaneously interacting over the internet, not setting out to write articles. You're "eavesdropping," here, and so if the content doesn't seem tailored to you or your better understanding of, it's because it wasn't actually intended for your consumption.
- 1317 2y ago
  then why was it posted here, in a public forum for the discussion of (technical) articles?
  - meepmorp 2y ago
    Because other people here might find it interesting. The world isn't geared towards you.

jetbalsa 2y ago

That is really interesting! I always though those things where solid state and wouldn't age over time! It makes a ton more sense that there is a somewhat of a reset that is randomly happening when people are fixing stuff nearby and thinking their main target was the thing that fixed it.

fuzzfactor 2y ago
Keep in mind that aluminum electrolytic capacitors are not solid-state devices themselves, actually liquid-state in a somewhat similar way to lead-acid batteries.
falcolas 2y ago
> I always though those things where solid state and wouldn't age over time! FWIW, thanks to a number of issues, even solid state devices will fail over time. https://c3.ndc.nasa.gov/dashlink/static/media/other/ObservedFailures37.html https://c3.ndc.nasa.gov/dashlink/static/media/other/Observed... (This entire presentation is worth the read, by the way).
lxgr 2y ago
Many solid state things age: Plastic and rubber can get weird/gross to the touch over the course of a few years or change color (due to additive breakdown/outgassing, UV radiation etc.). Flash storage cells can lose their stored information (over time) or their capacity to retain it (due to wear, i.e. erase cycles that are each slightly destructive to the materials) etc – and we even use them in things called "solid state drives" :)
kjs3 2y ago
things where solid state and wouldn't age over time Common misconception. Most solid state tech will fail over time, with wildly different timescales depending on the tech. For example, semiconductor technology is subject to electromigration, thermal effects and other material failures. As design geometries get smaller, these effects get worse.

nubinetwork 2y ago

It's not the capacitors in 360s that failed... it's something in the GPU, whether it's the interconnect, the die, or bad underfill, people have been trying to figure it out for years... but the failure analysis tools are expensive, and Microsoft will never come clean on why they failed...

jetbalsa 2y ago
He is making a good bit of sense, if it was the tiny 0402 SMD Caps near the GPU going south that would explain a good bit of what is going on
- NikkiA 2y ago
  Ceramic capacitors rarely die, when people talk about recapping electronics it's the electrolytic capacitors that they're talking about; which dry out.
  - robin_reala 2y ago
    Not dying, but falling out of tolerance specification.
    - eqvinox 2y ago
      That's equally rare for MLCCs, in particular for mass decoupling caps. It's really the Al-electrolytics and Tantalums where this happens.
phendrenad2 2y ago
The prevailing theory back when I was in the scene was that the GPU was warping the PCB itself, causing solder balls to lose connection. Heating in an oven softens the entire board, allowing it to un-torque.
- loa_in_ 2y ago
  My theory was that high operating temperature caused the balling to get partially softened and develop crystalline defects when cooling. I base it on personal experience that they never failed while running, but at least three people I know experienced these failures after powering down PCs for few days. All four GPUs were fixed by a friend of mine by... cooking them in the oven briefly

hellojebus 2y ago

I used to be local teen that would fix everyones Playstation 3's, when they got the red light of death. I used to do the heat gun trick to get them back up, but decided to also try an oven at one point (it worked). I wonder if this was the same phenomenon...

nubinetwork 2y ago
My understanding is that the ps3 and 360 had similar (but not entirely identical) issues, but the common "repair" on the ps3 was to change some unique (NEC tokin) capacitors around the CPU/GPU... only problem was that, like the hotbox "repair" it would fail again within a month if it worked at all.
sebazzz 2y ago
I fixed my NVIDIA video card by putting it in the oven. I wonder if I fixed the capacitors instead of the solder.
- spacebacon 2y ago
  I’ve exhausted options on my 4090. I sense a need for a “can you bake it” YT channel.

jauntywundrkind 2y ago

Long post, so sharing the key part, > if you heat the ceramic [capacitor] above its Curie temperature, all of the titanium atoms go back into a regular cubic lattice. if you cool it rapidly (just heat sinking away to the board is fine) it resets the aging process. and you can do this infinity times. suddenly the capacitance is back up to where it was before it even left the factory. > the Curie temperature is pretty low. like, 120°C ish. https://chaos.social/@gsuberland/113084484599859593 https://chaos.social/@gsuberland/113084484599859593 Worth reading it all!

renewiltord 2y ago

The nVidia G84 series back in the late 2000s had this problem and that was definitely the BGA solder. They reissued the GPUs after fixing.

fuzzfactor 2y ago

As a message to future visitors who may think "audio" capacitors; For those who are using ceramic coupling caps in their audio circuits, this can help explain some of the differences that can be detected between different types of caps. Especially in vintage-style vacuum tube assemblies where I would expect sometimes the internal temperature does rise above 120 C, and this would depend on ambient temperature too.

mystified5016 2y ago
Fun fact, ceramic capacitors (MLCC in particular) have microphonic effects. Vibration and mechanical strain in a capacitor generate a very slight voltage.

paulgerhardt 2y ago

The author is conflating a different contemporaneous issue of the PS3 (YLOD) with the XBox 360. It wasn't "capacitors". Further, I would argue it wasn't monocausal but a big finger can be pointed at thermal cycling fatigue. In broad strokes, a number of issues caused the Red Ring of Death ("general hardware failure") but the most dominant/infamous one was eventually attributed to a cross-domain electro-mechanical issue causing connections to break in the GPU interposer from thermal cycling in the first generation xbox gpu chips (made at a 90nm process node) which were rushed to market and fixed with second generation chips (made at 65nm process node)[1]. Initial attempts to individually address the solder, heat delta (by reducing the max temperture), and mechanical design of these chips did not fix the issue. Ultimately all three had to be sorted. The "towel trick" worked by making the resin in the PCB temporarily more pliant - allowing the x-clamp to reseat the gpu and temporarily re-bridging these broken connections on the GPU and to a lesser extant the CPU. Similar idea when swapping out a stronger "x-clamp" - aftermarket or with washers. Old leadership was given two shots to fix this initially and failed with Zephyr[early 2007] and Falcon[late 2007] generations which were best efforts given the cruel reality that GPU architecture turnaround time of around 2 to 3 years. Ultimately the fix had to come from three sources: ATI (new chips), replacing Peter Moore with Don Mattrick (new leadership and comms), and new hardware revision (Jasper[2008]). Earlier attempts to "replace the lead-free solder with a different formulation" in 2007 were ineffective in addressing the problem. Fundamentally it was a calculated gamble - Moore got a huge (albeit short-sighted) win for himself and Microsoft doubled their game console market share by pushing their vendors and themselves to compromise their release engineering schedule. It's debatable if they would have captured more share by running a more diligent process - probably no - given the issues were initially latent and didn't manifest for a few years, but certainly they would have lost less market share later on in 2008 and 2009. They couldn't, for instance, have known the PS3 would be so expensive at launch nor that the Wii would be such a sleeper hit in an untapped market segment. [1] https://www.youtube.com/watch?v=z2d6IMBS8oY&t=1130s https://www.youtube.com/watch?v=z2d6IMBS8oY&t=1130s

deleted 2y ago
[deleted]
krackers 2y ago

chomp 2y ago

I don’t think this post is accurate, there were multiple problems with the 360 design, it wasn’t capacitors. The towel trick warped the board and sometimes it worked and most of the time you wound up with a warped board and a system that continues to RROD. I fixed mine this way and still have it at my parent’s house, I’ll take a picture when I can, it’s very obvious what happens with the towel trick.

fuzzfactor 2y ago

With solder in the title this might be a good time to remember how much less of a conductor solder is compared to copper. This also may not apply very much to gaming consoles in particular. If the wire carrying high current has copper that is too thin gage, it will heat to some extent since its resistance is not quite zero, and at any one current the more resistance then the more heat will need to be dissipated. If there is a point where it is not pure copper from end-to-end but has a solder joint in between, the solder will be the point of highest resistance so it will autoheat to a higher temperature than the copper does. Plus when a solder joint cools properly it solidifies as a eutectic alloy for the most part rather than a crystalline solid. A "cold" solder joint can have more of a crystalline nature, plus can have lots of metal oxides included if it was allowed to remain molten for too long or at much higher temperature than necessary considering its melting point. IOW a cold solder joint can be even less of a conductor than normal, so much less it can even sometimes act as a bit of a ~semi~conductor in exactly the spot you would least want that.

mystified5016 2y ago
Resistance is a function of cross-sectional area of a conductor. Current capacity is a function of resistance, length, and heat dissipation (largely surface area). The cross-sectional area of a copper trace is on the order of ten micrometers square. The cross-sectional area of a solder joint is usually on the order of one or two square millimeters, if not more. For a BGA ball, you can approximate it as a cylinder with radius equal to the ball radius. Again, this is on the scale of millimeters. No matter what, your solder joint will have an order of magnitude larger area than the trace feeding it. Solder joints generally have far lower resistance than a copper trace due to this fact. Besides that, many (most?) lead-free solder alloys use silver, which is famously quite a remarkable conductor. Cold joints and the other problems you mention are basically irrelevant electrically. They're mechanical problems that are likely to cause a break in the circuit. Any added resistance from inclusions or crystal structure are completely irrelevant unless you're doing something extremely specialized like high sensitivity metrology or crazy RF stuff.

somat 2y ago

On the other side of the balance are the "it has nothing to do with the capaciters it is the solder" crowd. https://www.youtube.com/watch?v=I0UMG3iVYZI https://www.youtube.com/watch?v=I0UMG3iVYZI (ripfelix) It is a long watch, but it goes into a fairly deep dive into the whole affair. The author came to the conclusion that while a small ratio of the problem was the caps or the bga, There was a new process in play where chiplets were bonded to a base(bumps) which was then bonded to the motherboard(bga). And all processing units made with this first generations bonding process had a higher then expected failure rate, That is, the ps3, the xbox360 and nvidea graphics cards of this generation are all failing for the same reason. Having said that, That post is an amazing deep dive into the chemistry of capacitors.