9 ms·

I have a pair of "identical" GPUs which have dramatically different performance. Swapping the order didn't help. It turns out it was because one was blowing hot

by fyp 7y ago

I have a pair of "identical" GPUs which have dramatically different performance. Swapping the order didn't help. It turns out it was because one was blowing hot air into the other. Sigh.

ddingus 7y ago
Damn. When I trip up on that kind of thing, I always just wonder at how absurd it is. Two modern miracles. Insane complex and capable tech. Simple. Hot. Air. Basics. Oh well, maybe laugh. I would.
- pacaro 7y ago
  Seymour Cray referred to himself as an "overpaid plumber" for good reason. Heat management has been a critical element of high performance computing for a long time
  - tonyarkles 7y ago
    If I recall, one iteration of their products sprayed a shower of cooled liquid Fluorinert[1] directly into the processor cards, with no heatsinks at all. [2] [1] https://en.m.wikipedia.org/wiki/Fluorinert https://en.m.wikipedia.org/wiki/Fluorinert [2] https://en.m.wikipedia.org/wiki/Cray-2 https://en.m.wikipedia.org/wiki/Cray-2
    - Baeocystin 7y ago
      It was fully immersed! The Fluorinert was circulated through the stacked cards, then re-chilled with a cold water heat exchanger.
o-__-o 7y ago
I laughed so hard at this and then realized this was probably the source of my SLI woes way back in the day. But for real, I recently replaced the thermal paste on four 10 year old servers and performance improved by 20%!! A full core of performance by ensuring proper heat transfer
- hinkley 7y ago
  Does thermal paste age well? I wonder how much if that was bad initial application vs the ravages if time.
  - iforgotpassword 7y ago
    It probably depends on the type of paste, but I've found completely dried out paste in old systems that would just fall into pieces when you touch it.
  - rwmj 7y ago
    AMD gave us a dual socket test system (back in the Phenom days) which started failing after a couple of years of very heavy use for builds, CI, etc. After much head scratching I eventually found that the thermal paste had dried out, and simply reattaching the CPU fans with new paste fixed the system and it lived for quite a few more years after that.
    - mcv 7y ago
      That is excellent advice. I've had machines that started out pretty quiet but got noisier as they got older. I figured it was gunk accumulating on the fans or other parts, but I guess checking the paste is a good idea too.
  - o-__-o 7y ago
    depends on the paste, but safe to say 10 years is probably beyond the duty cycle. its $10 for a tube of arctic silver (I went with Arctic's organic, non conductive one I forget the name) that you can use for 10 or more applications. it takes 5 minutes to replace the thermal paste on a cpu. remember intel cpus will simply throttle themselves at high temps, so replacing the paste will allow heat dissipation across all the cores (as I witnessed first hand)
satori99 7y ago
I have seen this happen before. In the 1990's I was an audio engineer for a radio station that had just spent millions on digital suites with hard drives that would crash regularly, destroying in-progress work. The station actually got techs to fly to Australia to try to diagnose the problem -- A bunch of SCSI disks stacked vertically was causing the topmost drive to overheat.
- toolslive 7y ago
  Had a similar experience with a JBOD chassis. The vibrations of the spinning drives caused resonance on some positions. The cure was to attach a patch of duct tape with a washer at a certain point on the chassis.
  - iforgotpassword 7y ago
    Heh I remember in the late 90s I put some rubber between the hdds and case to greatly reduce the noise of my desktop PC.
    - the-dude 7y ago
      I have seen rubber 'sleeves' for hdds
      - java-man 7y ago
        I don't understand why this isn't a standard practice.
        mnw21cam 7y ago
        That sounds like a really easy way to make a hard drive overheat. Way back, I had a 4GB SCSI hard drive that was extra tall, extra fast, extra noisy, and extra hot. I constructed a thick rubber box around it to try noise-proof it a little, but I also had to strap a CPU cooler to it and have the airflow enter the box and exit at a hole in the box after wrapping all the way round the hard drive. It worked. But just wrapping it in rubber would have been a very quick way to cook the drive. This is a drive that if just left running bare on a table would get too hot to touch. Hard drives produce a heck of a lot less heat these days.
tyingq 7y ago
That was probablematic in the heyday of all the various Unix vendors. It was pretty common for some machines to vent front to back, others back to front, or large differences in depth, etc. So if you weren't paying attention, you could easily make a multi-stage heater stack in a single rack. Probably still an issue today, though there's a lot fewer vendors and variation.
- tallanvor 7y ago
  Old supercomputers - those built before the days of rows of racks - had such character, though! I worked at PNNL in the late 90s, and while they did have newer supercomputers that had a more traditional rack design, they also had a number of older ones that were obviously designed to be seen as well as used.

by fyp 7y ago

I have a pair of "identical" GPUs which have dramatically different performance. Swapping the order didn't help. It turns out it was because one was blowing hot air into the other. Sigh.

ddingus 7y ago

Damn. When I trip up on that kind of thing, I always just wonder at how absurd it is. Two modern miracles. Insane complex and capable tech. Simple. Hot. Air. Basics. Oh well, maybe laugh. I would.

pacaro 7y ago
Seymour Cray referred to himself as an "overpaid plumber" for good reason. Heat management has been a critical element of high performance computing for a long time
- tonyarkles 7y ago
  If I recall, one iteration of their products sprayed a shower of cooled liquid Fluorinert[1] directly into the processor cards, with no heatsinks at all. [2] [1] https://en.m.wikipedia.org/wiki/Fluorinert https://en.m.wikipedia.org/wiki/Fluorinert [2] https://en.m.wikipedia.org/wiki/Cray-2 https://en.m.wikipedia.org/wiki/Cray-2
  - Baeocystin 7y ago
    It was fully immersed! The Fluorinert was circulated through the stacked cards, then re-chilled with a cold water heat exchanger.

o-__-o 7y ago

I laughed so hard at this and then realized this was probably the source of my SLI woes way back in the day. But for real, I recently replaced the thermal paste on four 10 year old servers and performance improved by 20%!! A full core of performance by ensuring proper heat transfer

hinkley 7y ago
Does thermal paste age well? I wonder how much if that was bad initial application vs the ravages if time.
- iforgotpassword 7y ago
  It probably depends on the type of paste, but I've found completely dried out paste in old systems that would just fall into pieces when you touch it.
- rwmj 7y ago
  AMD gave us a dual socket test system (back in the Phenom days) which started failing after a couple of years of very heavy use for builds, CI, etc. After much head scratching I eventually found that the thermal paste had dried out, and simply reattaching the CPU fans with new paste fixed the system and it lived for quite a few more years after that.
  - mcv 7y ago
    That is excellent advice. I've had machines that started out pretty quiet but got noisier as they got older. I figured it was gunk accumulating on the fans or other parts, but I guess checking the paste is a good idea too.
- o-__-o 7y ago
  depends on the paste, but safe to say 10 years is probably beyond the duty cycle. its $10 for a tube of arctic silver (I went with Arctic's organic, non conductive one I forget the name) that you can use for 10 or more applications. it takes 5 minutes to replace the thermal paste on a cpu. remember intel cpus will simply throttle themselves at high temps, so replacing the paste will allow heat dissipation across all the cores (as I witnessed first hand)

satori99 7y ago

I have seen this happen before. In the 1990's I was an audio engineer for a radio station that had just spent millions on digital suites with hard drives that would crash regularly, destroying in-progress work. The station actually got techs to fly to Australia to try to diagnose the problem -- A bunch of SCSI disks stacked vertically was causing the topmost drive to overheat.

toolslive 7y ago
Had a similar experience with a JBOD chassis. The vibrations of the spinning drives caused resonance on some positions. The cure was to attach a patch of duct tape with a washer at a certain point on the chassis.
- iforgotpassword 7y ago
  Heh I remember in the late 90s I put some rubber between the hdds and case to greatly reduce the noise of my desktop PC.
  - the-dude 7y ago
    I have seen rubber 'sleeves' for hdds
    - java-man 7y ago
      I don't understand why this isn't a standard practice.
      - mnw21cam 7y ago
        That sounds like a really easy way to make a hard drive overheat. Way back, I had a 4GB SCSI hard drive that was extra tall, extra fast, extra noisy, and extra hot. I constructed a thick rubber box around it to try noise-proof it a little, but I also had to strap a CPU cooler to it and have the airflow enter the box and exit at a hole in the box after wrapping all the way round the hard drive. It worked. But just wrapping it in rubber would have been a very quick way to cook the drive. This is a drive that if just left running bare on a table would get too hot to touch. Hard drives produce a heck of a lot less heat these days.

tyingq 7y ago

That was probablematic in the heyday of all the various Unix vendors. It was pretty common for some machines to vent front to back, others back to front, or large differences in depth, etc. So if you weren't paying attention, you could easily make a multi-stage heater stack in a single rack. Probably still an issue today, though there's a lot fewer vendors and variation.

tallanvor 7y ago
Old supercomputers - those built before the days of rows of racks - had such character, though! I worked at PNNL in the late 90s, and while they did have newer supercomputers that had a more traditional rack design, they also had a number of older ones that were obviously designed to be seen as well as used.