16 ms·
Building a $5k ML Workstation with Tiitan RTX and Ryzen ThreadRipper [video]
- a2h 6y agoInteresting video, thanks for sharing. Just curious if you have one with tests or benchmarks for the completed build and/or temps at high loads? Would be cool to see :)
- jeffheaton 6y agoThose will be coming!
- paol 6y agoIt's worth noting that if your ML work is entirely CUDA based (as often happens), you likely won't benefit from a Threadripper CPU. Downgrading to a Ryzen 9 or even 7 will reduce costs by a good bit. The savings can be pocketed or put toward a second Titan RTX + NVLink (48Gb usable VRAM).
- colincooke 6y agoShould note (from someone who has a few of these systems at my lab) unfortunately the consumer RTX cards don't do memory pooling. This means that although NVLINK is good for inter-GPU comms it doesn't actually allow you to run giant models that need the entire 48GB of memory for a backwards pass (treat the combined cards as "one card"). Not typically a problem for most people but worth mentioning
- WrtCdEvrydy 6y agoYeah, Quadros ... the cocaine of the ML world.
- colincooke 6y agoI think the nice Volta cards (V100) does it "properly". But out of reach for most small scale setups (academic labs, prosumer, independent researcher, etc.). Unfortunately the best case for high-mem use-cases is to just rent from GCP.
- paol 6y agoFrom https://www.nvidia.com/en-us/deep-learning-ai/products/titan-rtx/ https://www.nvidia.com/en-us/deep-learning-ai/products/titan...: "NVIDIA TITAN RTX NVLink Bridge The TITAN RTX NVLink™ bridge connects two TITAN RTX cards together over a 100 GB/s interface. The result is an effective doubling of memory capacity to 48 GB, so that you can train neural networks faster, process even larger datasets, and work with some of the biggest rendering models."
- deleted 6y ago[deleted]
- colincooke 6y agoYeah you're not wrong, but it's a bit misleading. This allows you to run faster, but it does it by allowing you to use a larger batch size (arguably not best practice but your mileage will vary). Memory pooling is a bit different in that you can treat the combined cards as a single card from TF/pytorch.
- ivalm 6y agoBut batch size is prob least problem since you can do data parallelism (send half batch to each gpu, combine on cpu). I think only model bigger than gpu mem is where you really wish for nvlink on v100s.
- p1esk 6y agoNone of the ML frameworks support memory pooling so unless you write cuda code yourself this point is moot.
- sabalaba 6y agoMemory pooling is irrelevant for DL training. 24 GB is enough to run batch size of 1 for Bert-Large so honestly this is a good choice. Some folks are saying that 2x 2080 Tis would have been better and that's true if you're doing convnets but any large scale language model fine-tuning you'll want to have at least 24 GB of vRAM.
- p1esk 6y agoYou contradict yourself. Memory pooling is precisely what would allow you to train your bert large on two 2080ti.
- sabalaba 6y agoNo, my comment says that the two 2080 Tis would be better for convnets / situations where you don’t need to train Bert-Large. If you’re sure about memory pooling looking working for DL, please share code and examples, we would love to see one.
- topspin 6y agoYes, thats about $1000 savings. Also, the 80+ Gold power supply is an inefficient choice given then lack of a second GPU; without the second Titan that 1000W power supply will never see 50% load. If you're over-buying power supplies for future expansion then use an efficient titanium rated supply which will waste less power at low loads. The price difference is $80.
- Alupis 6y agoI thought the 80+ Certifications were about how efficient the PSU was at not converting electricity into heat, ie. loss? Perhaps I was wrong?
- freeqaz 6y agoIIRC PC power supplies are most efficient at around 80% utilization. Below that they are not able to hit their "rates" efficiencies.
- Alupis 6y agoHmm, interesting. I've always oversized by PSU's as a matter of course, since I've always thought working at 60% capacity is better than 90% or whatever. I usually drop a 750 watt 80+ Gold into most of my builds, even though a 500 watt or even a 450 watt would be sufficient with a single GPU, and have no plans for a second GPU.
- paulmd 6y agoaiming for 50-60% capacity during typical operation is the standard recommendation. Efficiency usually starts tapering off below 50% and below 30% it falls off a cliff - however, that just means instead of an ideal 10W power consumption you're actually pulling 30W or something like that, it is usually not a big deal in absolute terms. (there are also some exceptions, some of the platium/titanium PSUs actually can hold pretty decent efficiencies right down into the basement.) 750W is a good "standard" recommendation, that's enough for any one GPU on the market. The rule of thumb is really more to guide people not to buy 1600W or 2000W monster PSUs just because "bigger number is better!". (Although those giant PSUs do have the advantage that they can often run completely passively under load, they won't kick fans on until 50% or 60% load, which for a 1600W PSU means you can comfortably run a high-end GPU and a high-end CPU completely passively.)
- confuseshrink 6y agoIt depends on how intensive your pre-processing pipeline is. With a really fast accelerator you can quite easily start to be bottlenecked by your CPU.
- paol 6y agoTrue, but Threadrippers start at 24 cores and go up from there. That's got to be some intense pre-processing. Not impossible I'm sure, but it would be unusual.
- paulmd 6y agoThreadripper is the only way to get more than the standard 20 PCIe lanes (and really only 16 lanes to the slots, on all but one board). It's possible that OP would have gone with a lower core count version if one existed, but the minimum buy-in on Threadripper 3000 series is the 24 core model. tbh this is kind of one of the ideal use-cases for Epyc. And with the way AMD has set up their pricing, it's actually no longer cheaper to use the workstation processors, in some situations it's significantly more expensive, they are really ripping you for the clock speed, and removing a bunch of other features in the process (RDIMM/LRDIMM support, etc). I strongly encourage everyone doing homelab and home ML rigs and similar stuff to really think about whether they want Threadripper, bearing in mind that threadripper is often more expensive than Epyc. It's no longer an obvious choice that server processors are for servers and home users can only afford workstation, it is the other way around. AMD offers some low-core-count single-socket Epycs that are ideal for "lighting up the platform" tasks like this. Like, 7232P is a $450 processor and the 7402P is $1150. And they don't offer anything like that on Threadripper. They clock slower, sure, but they're not really using the CPU anyway. And that gets you a full 128 PCIe lanes, octochannel memory and RDIMM/LRDIMM support so they can stack in the memory. If they want to game on it in their spare time then sure, Threadripper is probably the way to go.
- p1esk 6y agoThreadripper Pro is an ideal processor (higher clocked Epyc). Unfortunately it's OEM only at this point.
- csdreamer7 6y agoMost Ryzen consumer motherboards have a limit of 128 gigs of RAM and 16-20 direct to the CPU pcie lanes. Is 128 gigs of ram and x8 pcie lanes for dual GPUs, a bottleneck for ML workloads? I can see the lanes not being an issue for the next gen Titans, that will likely use pcie 4.0, but that is months away. Asking as someone outside the ML field.
- paol 6y agoThe reduction from 16x to 8x PCIe lanes is usually not a bottleneck for ML. Still, it's always a good idea to benchmark and validate the configuration, especially if you're planning to spend a lot of money on a bunch of identical systems. As for RAM, only you can know how big your datasets are. But if you're training models on GPUs the bottleneck is almost certainly going to be GPU RAM, not system RAM.
- proverbialbunny 6y agoIn order the bottleneck is: gpu ram, cpu ram, then pci-e lanes. There is a big delay moving memory from ram to vram to run a task on the gpu, so much so that you'd be better off running the task on the cpu if you can't fit it all in the gpu, or are very clever in how data is buffered, which isn't an option for neural networks. Because of this, the pci-e lane is not saturated except when first sending the data to vram. PCI-E 3.0 x8 runs at 7880MB/s, so if your gpu has 16gb of vram, the difference between x8 and x16 is 1 second, when a task can typically take 8+ hours to complete.
- Alupis 6y ago> It's worth noting that if your ML work is entirely CUDA based (as often happens), you likely won't benefit from a Threadripper CPU Perhaps for the actual ML part, yes, but a ton of work must be done first to organize and filter the data, which is where all those cores would come in handy.
- ericd 6y agoI’ve found that it’s really nice for things like image augmentation, and running RL environments in parallel. But maybe I should be doing augmentation in Dali.
- dodobirdlord 6y agoA caveat is that if you’re going to use multiple GPUs it’s essential to get something like a Threadripper or a Xeon that has the pcie lanes to provide the full 16 lanes or at least 8 lanes to each GPU.
- p1esk 6y ago2x 2080ti would be faster than titan rtx, provide the same amount of memory, and would be cheaper.
- paol 6y ago> provide the same amount of memory Are you sure? The last time I checked the situation with NVLink memory pooling with 2080ti cards was very unclear.
- colincooke 6y agoUnfortunately multi-GPU training doesn't scale linearly yet [1] so it's often a better call to get a larger card then two smaller ones, at least for the single-model case. [1] https://github.com/keras-team/keras/issues/9204 https://github.com/keras-team/keras/issues/9204
- BadInformatics 6y ago(non-TF) Keras has notoriously bad multi-GPU support though (and was generally not well optimized. Case in point, the latest version just re-exports/forwards to tf.keras). Looking at something like https://lambdalabs.com/deep-learning/gpu-benchmarks https://lambdalabs.com/deep-learning/gpu-benchmarks or https://github.com/tensorpack/benchmarks/tree/master/other-wrappers https://github.com/tensorpack/benchmarks/tree/master/other-w..., multi-gpu scaling on 2080tis seems pretty darn close to linear. Plus, there are benefits to having more than one accelerator handy on a local workstation. For one, it's much easier to have multiple experiments running simultaneously or to run parallel training (e.g. hyperparameter search or RL episodes). Given that only the uber-expensive enterprise cards have proper virtualization/time sharing, trying this workflow on a Titan RTX will most likely be suboptimal unless you always run models that can make use of most of the memory and compute (no RNNs, no Neural ODEs, etc.)
- Sholmesy 6y agoLots of drawbacks with this approach: - More heat - More power consumption - More noise - The GPU memory isn't addressable as a single unit
- 6y ago
- brian_herman__ 6y agoHere is their list: PCPartPicker Part List: https://pcpartpicker.com/list/Jhyzcq https://pcpartpicker.com/list/Jhyzcq CPU: AMD Threadripper 3960X 3.8 GHz 24-Core Processor ($1348.00 @ Amazon) CPU Cooler: be quiet! Dark Rock Pro TR4 59.5 CFM CPU Cooler ($89.90 @ Amazon) Motherboard: MSI TRX40 PRO WIFI ATX sTRX4 Motherboard ($389.99 @ B&H) Memory: Corsair Vengeance RGB Pro 64 GB (4 x 16 GB) DDR4-3200 CL16 Memory ($329.99 @ Amazon) Storage: Sabrent Rocket 4.0 2 TB M.2-2280 NVME Solid State Drive ($399.98 @ Amazon) Video Card: NVIDIA TITAN RTX 24 GB Video Card ($2499.99 @ Newegg) Case: Corsair Crystal 570X RGB ATX Mid Tower Case ($179.99 @ B&H) Power Supply: Corsair RMx 1000 W 80+ Gold Certified Fully Modular ATX Power Supply ($204.99 @ Best Buy) Case Fan: Corsair LL120RGB LED 43.25 CFM 120 mm Fans 3-Pack ($120.99 @ Best Buy) Total: $5563.82 Prices include shipping, taxes, and discounts when available Generated by PCPartPicker 2020-07-15 11:13 EDT-0400
- p1esk 6y agoYou can spend half of the specified costs on every single one of the listed components with zero impact on your ML work productivity. $330 for 64gb of ram, really?
- el_oni 6y agoThe RGB makes ML models train faster
- GaryNumanVevo 6y agoComputers are multi-purpose machines
- fokinsean 6y ago$120 for 3 fans lmao
- jeffheaton 6y agoWell, it did include the RGB controller, if that makes you feel slightly better. :-)
- 6y ago
- Jestar342 6y agoWith the size of air-coolers these days, and how they all have integrated heat pipes, I'm beginning to wonder if we've crossed the distinction barrier with liquid-coolers. Holy moly is that a big heatsink.
- capableweb 6y agoTo be fair, not all air-coolers are of that size. Person who built the computer is probably overdoing most of the things, including the cooling, even if you're running it constantly. But also depends on where in the world you live and what the climate is like.
- gameswithgo 6y agoThat isn't overkill size for a 24 core cpu. However that particular heat sink is not great on threadrippers. It is probably ok for the 24 core TR because it doesn't have chiplets on the edges. The DarkRock TR model doesn't have heat pipes on the edge of the heat plate. They just made the heat plate wider, and it suffers on the 3990x for it. The Noctua equivalent is a better choice for these cpus. For the normal desktop cpus, I like the Dark Rock better. Cooling is the same as noctua but was quieter for me.
- mjayhn 6y agoYeah I actually stopped doing water cooling (usually Corsair AIOs) this year and went with this HSF just for ease of complexity, not that I ever had any problems with my AIOs. I should note it's WAY louder than I thought it'd be and it's made getting to my NVME drives a bit difficult. I didn't do much research beyond "best analog HSF vs watercoolers" and when it showed up I couldn't believe how big it was.
- whywhywhywhy 6y agoI went from AIO to air cooling quite recently after the pump on my AIO failed. Honestly not sure if I was just going from a bad AIO to a good cooler (smaller version of the Dark Rock Pro) but I actually found it to be quieter and the performance difference. Honestly the best bit though was the piece of mind that I don't have liquid inside my computer anymore. The idea of the pump leaking down onto my GPUs when I'm not around was stressful.
- andrewon 6y agoWhen he said training on Google colab took one day and on his computer took 20 mins, did he compare with google colab CPU? The difference seems too large.
- potiuper 6y agoPlease fix title Tiitan typo.
- neilv 6y agoIf you only have $1K or less to spend, and you don't already have a sufficient PC that you can upgrade with a big GPU... A non-Threadripper Ryzen, a big GPU, and a big PSU in a big case will go most of the way for most people, and leave you with an easy incremental upgrade path for bigger GPUs (or maybe add a second GPU). Slightly dated info for my current ML server, which is nicely quiet in my living room, thanks to Noctua: https://www.neilvandyke.org/machine-learning/ https://www.neilvandyke.org/machine-learning/ (Side note that's not in that page: I like to use older ThinkPads with transplanted vintage keyboards for my workstations, so I needed to make a separate box for the GPU. But life would be easiler, with a lot less juggling complexity, if I simply had the big GPU in my laptop rather.)
- disgruntledphd2 6y agoI recently bought a P73 thinkpad specced out like this, and it's great. However, putting a GPU and lots of RAM into a laptop makes it very, very heavy so it's worth thinking about if that's acceptable for you.
- 0xfaded 6y agoI just bought a specced out 1950x on a x399 with 64gb ram (plus the box, power, etc) for about $800, which I think is the fair price for 3 year old hardware. It needs a GPU, but for my usecase it's perfect. I'm also in Europe, so prices are higher.
- disgruntledphd2 6y agoYeah, me too. I spent a lot of money on this machine, but amortised over about five years (which is how long my last one lasted), it's acceptable (that's what I keep telling myself anyway).
- juped 6y agoThreadripper has its own socket type so I'd go with a cheaper or older one of those. Though I think third-gen threadripper is another socket entirely
- peterpost2 6y agoDid not expect a video that wholesome.
- highfrequency 6y agoThanks for the video! Could you comment on the differences between the Titan RTX and the V100? I am a bit confused because the V100 is significantly more expensive ($7k on Amazon even for the 16GB version) and has a slower clock speed, yet it is the standard in ML research papers. I see that it has ~10% more CUDA cores, but it doesn't seem like this would warrant a 3x price increase.
- freeqaz 6y agoIt's price segmentation. If you NEED the slight increase in power (and specific features like float8 at full speed) then NVIDIA charges significantly more for that. Gamers are more price sensitive than ML developers.
- Uehreka 6y agoThe other commenter mentioned the “pro-level” tradeoffs, but there’s something else too: Nvidia’s licensing won’t let you use GeForce cards in the cloud. If you’re building a datacenter, you have to use the Teslas.
- deleted 6y ago[deleted]
- smabie 6y agoIt's price discrimination. There's no reason why anyone would want a V100 besides that Nvidia doesn't "allow" you to use GeForce cards for ML research on servers, assuming you're big enough.
- DoctorOetker 6y agohow exactly is this enforced? I don't have a ML box yet, let alone a load of servers, but I am contemplating assembling my first rig for ML. If I buy (perhaps secondhand) some GPU, do I risk the thing refusing to work if it incorrectly thinks I'm a server farm? I have no idea how this could work, or is it just limited to 1 GPU per box? or the proprietary driver phones home? or certain CPU / mobo chipsets are detected and it refuses to run, even if its your only box?
- gameswithgo 6y agoIf you go with air cooling on a threadripper, I suggest going with a Noctua cooler instead of Dark Rock. Dark Rock extended the size of the heat plate to match the TR cpu size, but they didn't cover it with heat pipes, Noctua did. Cooling performance really suffers on the 3990X because there are chiplets on the edge of the cpu. the 32 and 24 core models it may not matter so much. See: https://www.kitguru.net/components/cooling/luke-hill/threadripper-3990x-cpu-cooling-comparison-how-to-tame-the-beast/11/ https://www.kitguru.net/components/cooling/luke-hill/threadr... On non threadripper cpus I actually like Dark Rock better. Cooling is the same as Noctua but it looks cooler and was quieter for me.
- trzeci 6y agoMy addition is that Dark Rock Pro TR4 is pretty bulky and I have a problem with Asus Zenith Extreme (Please note it's the older generation for 2950X) and radiator covers my PCIE#1 slot, so that I can't put graphics card there.
- fsociety 6y agoNoctura is even more bulky, unless you get the slim version.
- bob1029 6y agoNoctua is an automatic default for me now. I've got the NH-U14S on my 2950X and a NH-D15 on my 1800X. Never have any problems with these. Easy to install and maintain. Will probably reuse both when I upgrade my CPUs.
- bicknyers 6y agoAlso if you go air cooling and are confident with your abilities, consider delidding to drop temps. more (5 to 20C). If you go air cooling I would assume it is on the basis of long term stability, so don't use liquid metal either. Also invest in a nice PSU (gold minimum) with your peak load pulling only 75% of the rated max wattage Edit: Like most things look at components real-world testing figures, in this case, wattage, as opposed to TDP when planning
- odomojuli 6y agoI'm a bit concerned the build uses a Gold Certified power supply unit? Even for cheaper builds for non-ML workstations I would still only use Platinum and nothing less. I've been told Titanium is excessive but I mean I leave these things on for a while and power is expensive. For the DIY enthusiast or the WFH researcher, also the amount of heat involved can be a considerable cost in cooling or utility cost which varies substantially by floor of a building. It's probably not good, but not that bad to aircool this many GPUs as I've done in the past but it definitely means I'm paying a lot for A/C in the summer but almost nothing in the winter. I think Smerity even said he heated his small bedroom through the San Francisco winter off of one GPU while researching YOLO. Point: These things get hot. They require a lot of electricity. You should be concerned about a good PSU even for smaller builds. My energy cost for a 6GPU rig ran me about 1/3 of my total rent for a small apartment. That's electricity BEFORE I calculated my A/C bill which was separate and also substantial. My landlord hates me because I initially talked him into including it with my rent. All in all, it still makes sense to keep investing in local workstations, on-premises builds. No security concerns about a cloud, no futzing around with integrated notebooks, you own it you control it, and the price point up front is extremely attractive compared to base rates for cloud computing even on specialized hardware like a TPU. The numbers I come up with for batches still have a wide gap of several thousand USD most of the time, and then there's how much time it takes and how likely their service breaks. So kudos for the person who put in the effort to put this together and share. Any and all efforts towards making ML/DS affordable and DIY rises the tide for all boats. Question to the audience: Does anyone build GPU rigs like this for cryptocurrency anymore? I was only able to build a workstation once the price for GPU cards crashed.
- gameswithgo 6y agoThe efficiency delta between Gold and Titanium is really small. Optimizing that for heat reasons would be optimizing less than 1% of total heat output. And most cases keep the power supply thermally separate from the rest of the stuff anyway. This guy has a very oversized Gold power supply, the efficiency would be ~92% with gold vs 94% with platinum. Maybe a smaller titanium one would be a better overall choice I guess.
- 6y ago
- colordrops 6y agoBeing unfamiliar with ML work, when does it make sense to build one of these vs spinning up some instances on AWS or gcloud?
- bob1029 6y agoI think it really depends on how much you care about ML and how performant you actually need it to be. If you are a hobbyist or prototyping something speculatively for work, perhaps a cloud instance is prudent. If ML is your life's work, I'd probably consider throwing down for a proper rig so you don't get killed on cloud hosting fees.
- darknoon 6y agoNow is a particularly bad time to build a rig, since new NVIDIA cards are launching in a couple months. The value of a used 2080Ti (Turing) will tank, because Ampere cards will be available with similar performance for half the price.
- CarbyAu 6y agoAgreed. I need to update my gaming rig. Waiting for - Ryzen 3 - next round of GPUs from both vendors (although ML folks likely stay nVidia of course) - with luck, a better pcie4 SSD will be out by then too. I really wouldn't build one now unless I had to.
- svnpenn 6y agoWhat do people use ML for these days? I do computer programming, and I have done some work with video encoding, but this just seems like a huge investment money wise. So I am curious what use it is. For my needs the most intensive thing I do is compile some large programs or encode some large video, which you can get a computer for that for like $800.
- aunty_helen 6y agoHere's an example for what I'm using it for: https://news.ycombinator.com/item?id=23608360 https://news.ycombinator.com/item?id=23608360 I explained the technical details in the sub comment. I was looking at buying one of these Titan cards a few weeks back but then nvidia announced the next gen processors were coming out so have decided to wait until they refresh the 2 year old titan line instead of paying full prices for an almost out of date card. When training models for the object detection, the current algo we're using isn't focused on memory efficiency. So the 8gb card we currently use to train models is unable to process images at the correct resolution. We have to down scale about half to get it to fit. With the Titan RTX you get 22gb which is enough. On another note, the titan cards aren't the same as the normal geforce cards. Nvidia have gone to great lengths to ensure product differentiation so they can charge power users with business budgets more than people sitting at home playing games. One of the good things about the titan cards is they have a dual memory controller so you can write and read at the same time which improves your fill rate.
- proverbialbunny 6y agoML is typically used to find correlations in data. If something happens over and over again, there is a high chance it will happen again. Having such an algorithm that has identified this correlation allows it to identify when it will happen again. This allows for what is called predictive analytics. This can be as simple as identifying when a customer will end their service with a business, as there might be a pattern before previous customers have left, predicting when new customers are going to leave, and giving them a coupon or similar right before they would otherwise leave. This problem is called customer churn. It can be as complex as identifying when hardware will fail ahead of time, or even bio-ware. For example, I did a project that predicted when people were falling into depression before they could tell they were with a high accuracy rate. I also predicted other future medical issues ahead of time, like the probability an elderly person is going to fall over within the next handful of days. On the business side there are a lot of use cases for ML, but it falls more into analytics than engineering, as it's about predictive insight.
- mikece 6y agoDoes anyone do a measure of how long it would take such a workstation to pay for itself (including some nominal amount of operational cost for electricity) compared to simply doing ML on AWS/Azure/GCP? Seems like such a metric could be a useful measure for comparing such machines.
- CoolGuySteve 6y agoA comparable workstation costs about a month of on-demand EC2 time or 3 months of spot instance time. AWS GPU instances are really expensive. The most cost effective imo is to build a workstation for development and then deploy to AWS spot if you need a cluster. If you can't use a workstation for whatever reason, then use the new AWS feature to "stop" spot instances and use the spot instance as your workstation while being conscious of the high hourly cost and shutting it down when you're not working.
- FridgeSeal 6y agoAzure ML/GPU instanced are also really expensive. I did the maths recently and figured out I could put together a machine with a couple of 2080 Ti’s and have it pay for itself in a couple of months. I’m very seriously considering doing it, especially as I’m the only data scientist, if I had a team I’d be more in favour of going to the effort of setting up cloud-based training jobs etc
- CoolGuySteve 6y agoThat's what my partners and I did. But we bought refurbished 1080 cards for about $300 each and Ryzen 9 hardware. We're waiting for the 3000 series to come out which should be a large performance/dollar improvement over the current gen cards due to the smaller transistor size.
- m0zg 6y agoHere's my recommendation (I've built several such machines for my own use): 1. Go with a 1600W PSU from EVGA or Corsair. Other brands are hit or miss if you ever need very high current on the rails. This will manifest in your machine suddenly powering off when all 4 GPUs are hit with data at once (as is typical at the start of an epoch) 2. Use a mobo with evenly spaced GPU slots, such as ASRock TRX40 Creator. That way you can install 4 GPUs eventually and use that 1600W PSU. You also get 10GbE for distributed training, which is nice. 3. Don't waste money on Titan RTX, get 2x2080ti's instead. Then after a while get two more. Buy blower cards which blow hot air _out_ of the case. 4. Use an extension cable to install SSD and do not install it under a GPU - it'll die eventually due to overheating. 5. Air cooling is fine 6. If you have more than 2 GPUs learn how to adjust fan speeds on GPUs. Crank them to 85-100% while training to prevent throttling.
- sabalaba 6y agoGood choice on the 24 GB Titan RTX (so you can do at least batch size = 1 for Bert-Large). Not sure if that's the reason it was chosen though to be honest. If you want to do convnets only then you would do better with NVLinke'd 2080 Tis. Secondarily, I would suggest that you guys not use windows but instead Ubuntu 18.04 or 20.04 LTS and just install Lambda Stack (https://lambdalabs.com/lambda-stack-deep-learning-software https://lambdalabs.com/lambda-stack-deep-learning-software). It's a debian PPA that we maintain at Lambda to keep all of your deep learning drivers, CUDA, CuDNN, TensorFlow, and PyTorch up to date with just apt. It's free!
- mastazi 6y agoInteresting! Is Lambda Stack going to work on 20.04? The link mentions only 16.04 and 18.04
- hughdbrown 6y agoI've asked Lambda Labs 2-3 times if they are going to do an Ubuntu 20.04 stack, but I have not had a reply yet.
- mushufasa 6y agohow does that compare to the pop_os! nvidia drivers, default on their downstream-from-ubuntu distro?
- proverbialbunny 6y ago
- rcgorton 6y agoDid you consider NOT using NVidia? Both the capital cost and operational costs are huge compared to fairly high end Radeon cards (NO, I"m not an AMD employee - I merely choose based upon my personal budget)
- dodo6502 6y agoI think that tape-like piece that you removed from the SSD compartment is actually the thermal pad that makes contact between the SSD and the MSI heat sink cover so you may actually want that!
- zmmmmm 6y agoI am curious about the opposite end of the spectrum. What is the smallest and cheapest self contained setup that can be a serviceble development box for someone doing ML / AI type work? Does not need to run the production load, but has to be capable enough to allow local development activity that is still representative enough. So far the best I have identified is Intel NUC8 + nVidia GPU via Thunderbird. But it is still $1000 at least by the time you have it all together. NB: I know lots of people will say, just do it with cloud, but I work in a setting where much of my data cannot be put in the cloud, and also where the cost structure of funding well allows for fixed capital expenditure but not variable cloud costs.
- plasticchris 6y agoJust buy a case, motherboard, cpu, GPU, ram, psu, and build it. At the extreme low end you can buy a refurb Dell tower and drop in a new GPU.
- p1esk 6y agoThis entirely depends on the specific ML work you want to do. Smallest and cheapest could be something like Raspberry Pi or Jetson Nano. By the way, $5k ML workstation is still on the cheaper end of the spectrum. An 8x A100 machine will set you back at least $100k. And even that won't be enough to finetune GPT-3.
- fomine3 6y agoBuy used ATX tower desktop PC on Skylake gen (or buy new parts for Ryzen 3500 build), buy GPU (2070 SUPER for budget/perf?), buy new 750W PSU, put these parts. GPU via Tunderbolt looks like most expensive way.
- dzink 6y agoA bigger box helps reduce cooling and power expenses. I built a ThreadRipper tower with 2080TI last year and used BeQuiet 900 for it with very nice results.
- mpfundstein 6y agoi have a threadripper 1920x with 2x2080ti. When running cpuburn, i get around 65 tdie temp and with gpuburn, the upper card gets to around 86 and the lower one to 81. i have right now a water cooler for the cpu, 3 inlet fans (bottom back) and 2 outlet fans through the water cooler on top. i was wondering what temperatures I should aim for and what an optimal fan configuration is. i have a couple of fans laying around. the case is lian li O11 Air and the mobo is a taichi x399. anyone any tips? also I would want to use SLI. but then I would have to remove the fans on the gpu. Do I need then to water cool the gpu or what is the solution? if anyone of the moddibg pros here can help that would be awesome :-)