3 ms·

A self built machine with dual 4090s, soon to be 3x. Watercooled for quieter operation. Did the math on how much using runpod per day would be, and bought this

A self built machine with dual 4090s, soon to be 3x. Watercooled for quieter operation. Did the math on how much using runpod per day would be, and bought this setup instead. Using Fully sharded data parallel and bfloat16, I can train a 7b param model very slowly. But that’s fine for only going 2000 steps!

mrkeen 3y ago
> Watercooled for quieter operation. How is that these days? I water-cooled my system with the intention of having a quiet system sometime around 2008, but it just used liquid to transfer heat to a sink further away, which had a noisy fan.
- godelski 3y ago
  Much better these days and easier to do. Fans are aso much quieter these days too. A few years ago I was twisted into building a water cooled GPU system because it was the only 30 series card I could get in a 6 month period. Wasn't too hard to build into a system that doesn't even have enough room for the water pump to stand vertically. Soft tubing is annoying though and will fatigue with time, restricting flow rates. So I suggest if you do that to take more time to ensure you have angled fittings where you need them and to zip tie straight paths where you can (and thicker tubing is better). Might just be easier to go hardline instead tbh. Thermals are great though. But tbh now what I do is put my machine behind my TV and I have the side open and place the pump outside to stand vertically like old systems that had them outside the case lol. That's because if I'm not playing games on it then there's no difference in just sshing into the system because I live in the terminal anyways.
- buildbot 3y ago
  Pretty straightforward these days! Though I've been doing it for awhile. There a lot of little gotcha's like tubing choice and fittings but once you get the hang of the options it's pretty fun. Though cooling 4090s even with water is hard - you need a good flow rate to dissipate 450W. They'll hit near 70C during training if the fans are kept below audible, which works for me. 55-60C with fans at 50%. This is with 2x420mm (3x 140mm fan) rads and one 280mm rad (2x 140mm fans). The stock heatsinks actually were actually able to cool them just as well, just they where much louder doing so! Power is the other tricky bit. The computer is living on the dedicated AC circuit for now... 1600W is a lot. When it was on the single outlet circuit in my house, I could see each iteration as the lights dimmed in sync with the power draw surging.

mrkeen 3y ago

> Watercooled for quieter operation. How is that these days? I water-cooled my system with the intention of having a quiet system sometime around 2008, but it just used liquid to transfer heat to a sink further away, which had a noisy fan.

godelski 3y ago
Much better these days and easier to do. Fans are aso much quieter these days too. A few years ago I was twisted into building a water cooled GPU system because it was the only 30 series card I could get in a 6 month period. Wasn't too hard to build into a system that doesn't even have enough room for the water pump to stand vertically. Soft tubing is annoying though and will fatigue with time, restricting flow rates. So I suggest if you do that to take more time to ensure you have angled fittings where you need them and to zip tie straight paths where you can (and thicker tubing is better). Might just be easier to go hardline instead tbh. Thermals are great though. But tbh now what I do is put my machine behind my TV and I have the side open and place the pump outside to stand vertically like old systems that had them outside the case lol. That's because if I'm not playing games on it then there's no difference in just sshing into the system because I live in the terminal anyways.
buildbot 3y ago
Pretty straightforward these days! Though I've been doing it for awhile. There a lot of little gotcha's like tubing choice and fittings but once you get the hang of the options it's pretty fun. Though cooling 4090s even with water is hard - you need a good flow rate to dissipate 450W. They'll hit near 70C during training if the fans are kept below audible, which works for me. 55-60C with fans at 50%. This is with 2x420mm (3x 140mm fan) rads and one 280mm rad (2x 140mm fans). The stock heatsinks actually were actually able to cool them just as well, just they where much louder doing so! Power is the other tricky bit. The computer is living on the dedicated AC circuit for now... 1600W is a lot. When it was on the single outlet circuit in my house, I could see each iteration as the lights dimmed in sync with the power draw surging.