44 ms·

The author wasn't joking about the noise levels. This machine sounds like an F1 race car. If you don't require a rack mounted server, a cluster of workstations

The author wasn't joking about the noise levels. This machine sounds like an F1 race car. If you don't require a rack mounted server, a cluster of workstations like NVIDIA's DIGITS DevBox is far more cost efficient (and less noisy). I run a compute intensive business (Dreamscopeapp.com) and we opted to build a cluster of desktop-like machines instead of using a rack mounted solution. Another benefit is you don't run into the power issues mentioned in the post. My start-up actually sells the machine described in this post: https://lambdal.com/nvidia-gpu-server https://lambdal.com/nvidia-gpu-server And a machine inspired by the NVIDIA DIGITS DevBox: https://lambdal.com/nvidia-gpu-workstation-devbox https://lambdal.com/nvidia-gpu-workstation-devbox

highd 9y ago
I feel like you should be able to do better than $15K for $7.2K worth of graphics cards.
- mippie_moe 9y ago
  You can save a ton of money by building your own machine. The server we sell is packaged with software we wrote that makes administering significantly easier. We also provide technical support and even a limited amount of free machine learning consulting. The customers who purchase this server want a headache free solution and aren't as price sensitive as a lone researcher.
  - jjn2009 9y ago
    heres a part list to build a pc with 1080ti's: https://pcpartpicker.com/user/Jjn2009/saved/#view=McvMpg https://pcpartpicker.com/user/Jjn2009/saved/#view=McvMpg notice the custom part box accounts for two more GPUs, I'm not sure why the site doesn't let you add 4 to the GPU section. This setup ranges from $5250 with 4 GPU to $3240 with 1 GPU. You might want to bump up the PSU for 4 GPU its currently 1500 watts, which may or may not be enough at max load. The article shows a max of ~2800 watts with 8 GPUs
    - dgacmu 9y ago
      nice - btw, the Rosewill Tokamak 1500 from newegg is a way to save a few bucks on that build, though it's out of stock (they just had it on sale). It's also 80+ titanium.
      - p1esk 9y ago
        Actually, that's a pretty bad config: Mobo does not support 4x16 PCIe lanes (that's why they didn't want you to add 4 GPU cards). Mobo is limited to 64GB RAM. $520 for two 256GB 840 Pro SSDs? Seriously? Here's a better mobo: https://www.amazon.com/Motherboards-X99-E-WS-USB-3-1/dp/B00XUDLXJG/ https://www.amazon.com/Motherboards-X99-E-WS-USB-3-1/dp/B00X... Also, you can literally double the RAM for the same money: http://www.ebay.com/itm/128gb-DDR4-8-Crucial-16gb-DDR4-2400mhz/252958548288 http://www.ebay.com/itm/128gb-DDR4-8-Crucial-16gb-DDR4-2400m... (keep in mind that speed of RAM is irrelevant for DL tasks).
science404 9y ago
Can confirm that. Have this box (with 8 K80s) and we measured 90dB next to the box..
EvgeniyZh 9y ago
Doesn't the fact you get 8 PCIe lines per GPU affects performance?
- mippie_moe 9y ago
  With this setup, you get one GPU that runs with 16 lanes (and as you mentioned, three that run with 8). Bottleneck depends on the workload. If you're training a small/fast network, data bandwidth is a real problem. That being said, for most cases, a workstation build that provides every GPU with 16 lanes is far less cost effective.
kough 9y ago
I ask this out of deep curiosity and by no means with any intent to offend: how does (will?) dreamscopeapp.com make any money? (I don't have an Android device or I'd install it – I'm asking because it's entirely unapparent from your web presence.)
- mippie_moe 9y ago
  No offense taken! Dreamscope doesn't actually make money :) It's a little under break even. It brings in revenue through a $9.99/mo premium subscription, which gives customers higher resolution images.
  - kough 9y ago
    Ah – that's fairly reasonable! Thanks for the answer :)
dgacmu 9y ago
So - tried your quote form. The options are 4x 1080ti, 4x titan Xp, or 8x P100 --- but no 8x 1080ti? Or is the quote form wrong? 16.5K seems pretty reasonable for 8x 1080ti with a bit of profit for building it, but unreasonable for only 4x 1080ti. My home-built 4x1080ti box (without quite enough PCIe bandwidth, admittedly) is under $6k. I'm assuming/hoping there's an error there. :) Screenshot of the order form: https://www.dropbox.com/s/2nm00w1rd6du6ey/Screenshot%202017-06-07%2022.17.15.png?raw=1 https://www.dropbox.com/s/2nm00w1rd6du6ey/Screenshot%202017-... Oh, also - if I want a quote on both the big server and the little workstation I have to enter my contact info twice? Not particularly customer-friendly.
- p1esk 9y ago
  For quad GPU config you should look at dev box type option. It's $8,750 for machine with 4 1080Ti, 64GB of RAM, and 1TB SATA SSD. Quite a steep margin if you ask me, considering a 128GB RAM machine that you build yourself would cost you at most $5,700 (taxes included) if you get everything from Amazon, and probably under $5k if you're willing to shop around a little.
sunsu 9y ago
I purchased a couple of the machines from these guys for my team and had a great experience. Highly recommended if you have better ways to spend your time than building them yourself.
braindead_in 9y ago
Nice! Would you be supporting Volta GPUs when it is launched?
- mippie_moe 9y ago
  Yes!
intoverflow2 9y ago
>The author wasn't joking about the noise levels. This machine sounds like an F1 race car. One of the reasons 3D artists who are using GPU rendering generally go for liquid cooling (that and it makes the cards single slot) http://rawandrendered.com/Octane-Render-Hepta-GPU-Build http://rawandrendered.com/Octane-Render-Hepta-GPU-Build
nojvek 9y ago
How much do the devboxes go out for? Prices not mentioned.

highd 9y ago

I feel like you should be able to do better than $15K for $7.2K worth of graphics cards.

mippie_moe 9y ago
You can save a ton of money by building your own machine. The server we sell is packaged with software we wrote that makes administering significantly easier. We also provide technical support and even a limited amount of free machine learning consulting. The customers who purchase this server want a headache free solution and aren't as price sensitive as a lone researcher.
- jjn2009 9y ago
  heres a part list to build a pc with 1080ti's: https://pcpartpicker.com/user/Jjn2009/saved/#view=McvMpg https://pcpartpicker.com/user/Jjn2009/saved/#view=McvMpg notice the custom part box accounts for two more GPUs, I'm not sure why the site doesn't let you add 4 to the GPU section. This setup ranges from $5250 with 4 GPU to $3240 with 1 GPU. You might want to bump up the PSU for 4 GPU its currently 1500 watts, which may or may not be enough at max load. The article shows a max of ~2800 watts with 8 GPUs
  - dgacmu 9y ago
    nice - btw, the Rosewill Tokamak 1500 from newegg is a way to save a few bucks on that build, though it's out of stock (they just had it on sale). It's also 80+ titanium.
    - p1esk 9y ago
      Actually, that's a pretty bad config: Mobo does not support 4x16 PCIe lanes (that's why they didn't want you to add 4 GPU cards). Mobo is limited to 64GB RAM. $520 for two 256GB 840 Pro SSDs? Seriously? Here's a better mobo: https://www.amazon.com/Motherboards-X99-E-WS-USB-3-1/dp/B00XUDLXJG/ https://www.amazon.com/Motherboards-X99-E-WS-USB-3-1/dp/B00X... Also, you can literally double the RAM for the same money: http://www.ebay.com/itm/128gb-DDR4-8-Crucial-16gb-DDR4-2400mhz/252958548288 http://www.ebay.com/itm/128gb-DDR4-8-Crucial-16gb-DDR4-2400m... (keep in mind that speed of RAM is irrelevant for DL tasks).

science404 9y ago

Can confirm that. Have this box (with 8 K80s) and we measured 90dB next to the box..

EvgeniyZh 9y ago

Doesn't the fact you get 8 PCIe lines per GPU affects performance?

mippie_moe 9y ago
With this setup, you get one GPU that runs with 16 lanes (and as you mentioned, three that run with 8). Bottleneck depends on the workload. If you're training a small/fast network, data bandwidth is a real problem. That being said, for most cases, a workstation build that provides every GPU with 16 lanes is far less cost effective.

kough 9y ago

I ask this out of deep curiosity and by no means with any intent to offend: how does (will?) dreamscopeapp.com make any money? (I don't have an Android device or I'd install it – I'm asking because it's entirely unapparent from your web presence.)

mippie_moe 9y ago
No offense taken! Dreamscope doesn't actually make money :) It's a little under break even. It brings in revenue through a $9.99/mo premium subscription, which gives customers higher resolution images.
- kough 9y ago
  Ah – that's fairly reasonable! Thanks for the answer :)

dgacmu 9y ago

So - tried your quote form. The options are 4x 1080ti, 4x titan Xp, or 8x P100 --- but no 8x 1080ti? Or is the quote form wrong? 16.5K seems pretty reasonable for 8x 1080ti with a bit of profit for building it, but unreasonable for only 4x 1080ti. My home-built 4x1080ti box (without quite enough PCIe bandwidth, admittedly) is under $6k. I'm assuming/hoping there's an error there. :) Screenshot of the order form: https://www.dropbox.com/s/2nm00w1rd6du6ey/Screenshot%202017-06-07%2022.17.15.png?raw=1 https://www.dropbox.com/s/2nm00w1rd6du6ey/Screenshot%202017-... Oh, also - if I want a quote on both the big server and the little workstation I have to enter my contact info twice? Not particularly customer-friendly.

p1esk 9y ago
For quad GPU config you should look at dev box type option. It's $8,750 for machine with 4 1080Ti, 64GB of RAM, and 1TB SATA SSD. Quite a steep margin if you ask me, considering a 128GB RAM machine that you build yourself would cost you at most $5,700 (taxes included) if you get everything from Amazon, and probably under $5k if you're willing to shop around a little.

sunsu 9y ago

I purchased a couple of the machines from these guys for my team and had a great experience. Highly recommended if you have better ways to spend your time than building them yourself.

braindead_in 9y ago

Nice! Would you be supporting Volta GPUs when it is launched?

mippie_moe 9y ago
Yes!

intoverflow2 9y ago

>The author wasn't joking about the noise levels. This machine sounds like an F1 race car. One of the reasons 3D artists who are using GPU rendering generally go for liquid cooling (that and it makes the cards single slot) http://rawandrendered.com/Octane-Render-Hepta-GPU-Build http://rawandrendered.com/Octane-Render-Hepta-GPU-Build

nojvek 9y ago

How much do the devboxes go out for? Prices not mentioned.