7 ms·

I agree that for gaming it makes sense getting the higher end cards rather than dealing with SLI, but my use case is password cracking that doesn't require five

by nikcub 10y ago

I agree that for gaming it makes sense getting the higher end cards rather than dealing with SLI, but my use case is password cracking that doesn't require five 9's uptime. The GTX 1070 is 150W power consumption while the 1080 is 180W. Rather than get another case i'd just scale up the case size based on the number of cards - 2 cards ($100[0]) 4 cards ($200[1]), 6 cards ($400[2]) 1200W PSU $300[3], riser cards $10[4] and then motherboard + cpu + SSD about $200-250 so it works out $700 base for 2 GPU system, $900 base for 4 GPU system and $1400 for 6 GPU (2 PSU's) If you work that out with the card options the sweet spot is 4x 1070 GPU's for $1500. It really isn't worth spending $4k on cards to get equiv performance from Titan X's or 15-30% more performance with the 1080 (although power/performance is better). [0] http://www.coolermaster.com/case/mid-tower-cm690-series/cm693/ http://www.coolermaster.com/case/mid-tower-cm690-series/cm69... [1] http://www.corsair.com/en/obsidian-series-900d-super-tower-case http://www.corsair.com/en/obsidian-series-900d-super-tower-c... [2] http://www.miningrigs.net/?product=graymatter-6gpu-server-case http://www.miningrigs.net/?product=graymatter-6gpu-server-ca... [3] http://www.newegg.com/Product/Product.aspx?Item=N82E16817139039 http://www.newegg.com/Product/Product.aspx?Item=N82E16817139... [4] http://www.amazon.com/Fixable-Adapter-Flexible-Extension-Connector/dp/B00IMYODGS http://www.amazon.com/Fixable-Adapter-Flexible-Extension-Con...

dogma1138 10y ago
What motherboard can you buy for under 50 with 6 PCIeX16 (power) slots? The cheap "bitcoin" motherboard that go for about 50-75$(old intel express chipset) won't work in this case, not sure about the 1070 but the 980 (non-ti) and the Titan X do not want to work with any PCIe slot that isn't a true PCIeX16 power slot. The PCIe x1-4 to x6 riser converter boards do not work, even the "good" ones with a SATA power connector and an actual power PCB for some reason, my guess is that there is some actual sensing going on in the card itself. Unless you buy something like this http://www.amazon.com/ASRock-Motherboard-H81-PRO-BTC/dp/B00HEUIS2G?ie=UTF8&keywords=asrock%20bitcoin%20motherboard&qid=1465300259&ref_=sr_1_1&sr=8-1 http://www.amazon.com/ASRock-Motherboard-H81-PRO-BTC/dp/B00H... which which is i think the only non-server motherboard which is commonly available you aren't getting more than 4-4 PCIe slots, and even to get 4 (PCIe16) you need to spend about 200$ on a motherboard these days and the H81 chipset mobo's don't want to work with modern nVIDIA cards (or the other way around) with all but the main PCIe16 slot. Pretty much to get more than 2 PCIx16 slots these days you have to buy a good gaming Z170/Z97/X99 motherboard, that ain't cheap, nor are the CPU's that go into them. While the AMD CPU's might be a bit cheaper again the motherboards won't be. P.S. my personal password cracking setup at home is 4 5970's you could get them for about 100$ even a couple of years ago (today they are much cheaper 2 for 130 or b/o: http://goo.gl/yplISa http://goo.gl/yplISa) and for most hashes they go 40-50% of the perofmrnace of a Titan X (I do have 2 Titan X's in my personal gaming rig which i can use in a pinch). I got them setup on an ASROCK motherboard (h61 iirc) that was designed for bitcoin mining so it comes with 8 PCIE slots and powered converter ribbons connect the cards do it all of this sits in a al cheapo rackmount case of a 20 year old compaq server. But if you are looking for more scalable compute instance AWS CUDA instances are probably a better solution and I've been using them more and more lately.
- Bedon292 10y ago
  I have been wondering about the AWS GPU instances. They are something like $2.60 / hour. At what point do they make sense monetarily? You can build a machine like described here for ~$5000 total. Thats not that many hours of AWS time. So if anyone is doing any long running cracking, or machine learning, does it ever make sense? I am truly curious about what people's experience has been.
  - dogma1138 10y ago
    g2.8xlarge $0.7414 per Hour (linux pricing) it makes sense unless you are doing it for a "hobby", you just roll costs into each project which makes it much easier from an accounting POV :P
  - illumin8 10y ago
    If you buy Spot instances you can get them for ~85% discount off normal pricing. Your spot instances can be pre-empted, of course, but for password cracking this shouldn't really matter much.
  - ryanlol 10y ago
    >At what point do they make sense monetarily? For password cracking? Never, especially considering you can just go on insidepro or something and rent access to someones cluster.
    - dogma1138 10y ago
      Insidepro is a great resource, but you usually aren't going to get a VAT invoice to either bill your client for or deduct in your taxes ;) AWS is simple, it's not overly expensive for most password cracking ventures (e.g. password quality audit that certain organization like to do on their AD/LDAP) and more importantly you aren't giving away potentially very sensitive information to people that by default cannot be trusted. Inside pro is ok if you are going mostly non-legit / off the books to begin with, it's also ok for basic research on non-real world data, but never use it for actual real world data not to mention B2B services.
  - 10y ago
gtani 10y ago
You may want to look at hybrid/liquid cooling with an X99 motherboard e.g. 4 cards x 16 PCI-e lanes http://www.newegg.com/Product/Product.aspx?Item=N82E16813132263 http://www.newegg.com/Product/Product.aspx?Item=N82E16813132... (tho you'd have to cut holes in the case to bolt radiator fans onto)
- dogma1138 10y ago
  The problem is then you are spending about 2000$ on your "chassis"(case, cpu, motherboard, memory, cooling etc.) if you are in that situation then getting the more expensive cards (1080/1080ti or even the Titan w/e they'll name it now) is usually the only way to go because your chassis is now a major factor of the final price so you'll need to squeeze a much performance from it as possible.
  - semi-extrinsic 10y ago
    Yes, I can confirm this is the case (pun intended). I've been speccing out GPU machines for molecular simulation, and any way you slice it, the actual GPUs account for less than 50% of your total cost, as long as you're looking at GTX class cards. The other issue with 8 GPU machines, which tend to use dual CPU motherboards to have enough PCIe lanes, is that you pay a serious performance penalty for communication between GPUs attached to different CPUs (over QPI). It's so bad that I've seen several standalone 8 GPU systems using FDR Infiniband (with GPUdirect RDMA) to communicate internally on the machine. Which is of course bloody expensive. If it suits your workload, having two separate 4 GPU machines is better and cheaper than a single 8 GPU machine.
- fredophile 10y ago
  Can you actually get full performance out of that? I thought the bottleneck these days is the CPU which maxes out at 40 lanes at the high end. This only gets you 1x16 and 3x8 even if all the slots support 16 lanes.
yread 10y ago
Why do you need so much password cracking power? Do you forget it that often?
- rhaps0dy 10y ago
  lol :) I'm inclined towards the hypothesis of parent being a pentester and buying for their freelance business, or company.
alvern 10y ago
Thanks for posting this, I was looking at this type of build with CUDA before pascal was announced.

by nikcub 10y ago

dogma1138 10y ago

What motherboard can you buy for under 50 with 6 PCIeX16 (power) slots? The cheap "bitcoin" motherboard that go for about 50-75$(old intel express chipset) won't work in this case, not sure about the 1070 but the 980 (non-ti) and the Titan X do not want to work with any PCIe slot that isn't a true PCIeX16 power slot. The PCIe x1-4 to x6 riser converter boards do not work, even the "good" ones with a SATA power connector and an actual power PCB for some reason, my guess is that there is some actual sensing going on in the card itself. Unless you buy something like this http://www.amazon.com/ASRock-Motherboard-H81-PRO-BTC/dp/B00HEUIS2G?ie=UTF8&keywords=asrock%20bitcoin%20motherboard&qid=1465300259&ref_=sr_1_1&sr=8-1 http://www.amazon.com/ASRock-Motherboard-H81-PRO-BTC/dp/B00H... which which is i think the only non-server motherboard which is commonly available you aren't getting more than 4-4 PCIe slots, and even to get 4 (PCIe16) you need to spend about 200$ on a motherboard these days and the H81 chipset mobo's don't want to work with modern nVIDIA cards (or the other way around) with all but the main PCIe16 slot. Pretty much to get more than 2 PCIx16 slots these days you have to buy a good gaming Z170/Z97/X99 motherboard, that ain't cheap, nor are the CPU's that go into them. While the AMD CPU's might be a bit cheaper again the motherboards won't be. P.S. my personal password cracking setup at home is 4 5970's you could get them for about 100$ even a couple of years ago (today they are much cheaper 2 for 130 or b/o: http://goo.gl/yplISa http://goo.gl/yplISa) and for most hashes they go 40-50% of the perofmrnace of a Titan X (I do have 2 Titan X's in my personal gaming rig which i can use in a pinch). I got them setup on an ASROCK motherboard (h61 iirc) that was designed for bitcoin mining so it comes with 8 PCIE slots and powered converter ribbons connect the cards do it all of this sits in a al cheapo rackmount case of a 20 year old compaq server. But if you are looking for more scalable compute instance AWS CUDA instances are probably a better solution and I've been using them more and more lately.

Bedon292 10y ago
I have been wondering about the AWS GPU instances. They are something like $2.60 / hour. At what point do they make sense monetarily? You can build a machine like described here for ~$5000 total. Thats not that many hours of AWS time. So if anyone is doing any long running cracking, or machine learning, does it ever make sense? I am truly curious about what people's experience has been.
- dogma1138 10y ago
  g2.8xlarge $0.7414 per Hour (linux pricing) it makes sense unless you are doing it for a "hobby", you just roll costs into each project which makes it much easier from an accounting POV :P
- illumin8 10y ago
  If you buy Spot instances you can get them for ~85% discount off normal pricing. Your spot instances can be pre-empted, of course, but for password cracking this shouldn't really matter much.
- ryanlol 10y ago
  >At what point do they make sense monetarily? For password cracking? Never, especially considering you can just go on insidepro or something and rent access to someones cluster.
  - dogma1138 10y ago
    Insidepro is a great resource, but you usually aren't going to get a VAT invoice to either bill your client for or deduct in your taxes ;) AWS is simple, it's not overly expensive for most password cracking ventures (e.g. password quality audit that certain organization like to do on their AD/LDAP) and more importantly you aren't giving away potentially very sensitive information to people that by default cannot be trusted. Inside pro is ok if you are going mostly non-legit / off the books to begin with, it's also ok for basic research on non-real world data, but never use it for actual real world data not to mention B2B services.
- 10y ago

gtani 10y ago

You may want to look at hybrid/liquid cooling with an X99 motherboard e.g. 4 cards x 16 PCI-e lanes http://www.newegg.com/Product/Product.aspx?Item=N82E16813132263 http://www.newegg.com/Product/Product.aspx?Item=N82E16813132... (tho you'd have to cut holes in the case to bolt radiator fans onto)

dogma1138 10y ago
The problem is then you are spending about 2000$ on your "chassis"(case, cpu, motherboard, memory, cooling etc.) if you are in that situation then getting the more expensive cards (1080/1080ti or even the Titan w/e they'll name it now) is usually the only way to go because your chassis is now a major factor of the final price so you'll need to squeeze a much performance from it as possible.
- semi-extrinsic 10y ago
  Yes, I can confirm this is the case (pun intended). I've been speccing out GPU machines for molecular simulation, and any way you slice it, the actual GPUs account for less than 50% of your total cost, as long as you're looking at GTX class cards. The other issue with 8 GPU machines, which tend to use dual CPU motherboards to have enough PCIe lanes, is that you pay a serious performance penalty for communication between GPUs attached to different CPUs (over QPI). It's so bad that I've seen several standalone 8 GPU systems using FDR Infiniband (with GPUdirect RDMA) to communicate internally on the machine. Which is of course bloody expensive. If it suits your workload, having two separate 4 GPU machines is better and cheaper than a single 8 GPU machine.
fredophile 10y ago
Can you actually get full performance out of that? I thought the bottleneck these days is the CPU which maxes out at 40 lanes at the high end. This only gets you 1x16 and 3x8 even if all the slots support 16 lanes.

yread 10y ago

Why do you need so much password cracking power? Do you forget it that often?

rhaps0dy 10y ago
lol :) I'm inclined towards the hypothesis of parent being a pentester and buying for their freelance business, or company.

alvern 10y ago

Thanks for posting this, I was looking at this type of build with CUDA before pascal was announced.