6 ms·

If I am looking to buy a GPU chip for ML research: - What chip should I buy? - When should I buy? Should I wait for prices to drop? Will new and improved chips

If I am looking to buy a GPU chip for ML research: - What chip should I buy? - When should I buy? Should I wait for prices to drop? Will new and improved chips be released anytime soon? - What are the advantages / disadvantages of each chip (3060 vs 3080, Nvidia vs AMD)? Which chip is most cost-efficient? What are each chips' specialties (e.g. specific type of neural network, graphics vs compute)?

learndeeply 4y ago
This applies for all neural networks. Depending on how much money you're willing to spend, in descending order: DGX (computer with 8 A100s, $150,000), A100 (80GB, $15,000), A6000 ($5000), RTX 3090 ($1000).
- moyix 4y ago
  I just grabbed an A6000 for $3500 on eBay, so you can probably get a pretty decent deal on those now. They're pricey but IMO it's a great deal if you really need the VRAM (e.g. for training LLMs).
- hedora 4y ago
  I'd consider renting these, especially if you are just getting started: https://docs.paperspace.com/core/compute/machine-types/ https://docs.paperspace.com/core/compute/machine-types/ 1 x A100, 80GB is $3.19 / hour. 8 x A100, 80GB is about $25 / hour. They have much less expensive machines. I used to use them to run steam games, but now proton is just too darn good. Their low end machines are OK for CAD software that supports real-time raytracing.
  - asciimike 4y ago
    Lots of good options for cheap GPU clouds, including Paperspace (mentioned above), Coreweave, and Crusoe Cloud (crusoecloud.com). Crusoe Cloud's angle is that our GPUs are powered off otherwise wasted energy and are carbon-reducing; running one for a year provides an emissions reduction equivalent to taking one car off the road. Disclosure, I'm head of product @ Crusoe Cloud. Feel free to ping me at mike at crusoecloud dot com if you've got questions or feedback.
    - pbronez 4y ago
      Cool idea. Tried to join your waitlist but the form throws errors. Could be my security measures, but I’m on mobile safari, toggled off all content blockers and still had an error.
      - asciimike 4y ago
        Odd, I just tested on Safari mobile and it went through without issues. Mind sending me an email at mike at crusoecloud dot com and we can get you set up?
    - douto 4y ago
      How can this be carbon-reducing? That sounds like a dubious marketing claim.
      - asciimike 4y ago
        TL;DR: we run data centers on-site at oil wells and take natural gas that would otherwise be flared (it can't be economically transported as natural gas in a pipeline or turned into electricity and transported) and combust it completely. Methane is a significantly more potent greenhouse gas than CO2, so it ends up being a net reduction in emissions vs what's currently happening. https://www.crusoeenergy.com/digital-flare-mitigation https://www.crusoeenergy.com/digital-flare-mitigation has some more information.
        ip26 4y ago
        Is flare combustion more incomplete than your generation?
        brimwats 4y ago
        on the other hand, does that mean you're obligated to be in oil producing countries? how does the geopolitics of that work?
    - hedora 4y ago
      How do you carbon offset? We have a bunch of diseased pine trees. Assuming we don't have enough time to cut them down and burn them in the next 12 months, and that I am unscrupulous, I could sell you carbon credits through one of the bigger exchanges. You would never know, and it would be completely above board. John Oliver had a particularly depressing segment on this recently. (Totally off topic, but I'd love to biochar them instead. Using them for lumber and shipping them offsite are non-starters. Any ideas, anyone?)
  - lbotos 4y ago
    As someone who just fired off both paperspace boxes and coreweave to play around with stable diffusion, Paperspace limited me to some old janky hardware and never replied to my request to "unlock" the more modern cards. Coreweave let me get a modern card box first go round. I'd go with coreweave if you are a weekend hacker who just wants access to a lil bit of beefy GPU. YMMV.
  - cwkoss 4y ago
    Are there "mining pools" for running AI jobs and getting paid in crypto?
    - knicholes 4y ago
      There's vast.ai, but you get paid in USD.
- UncleOxidant 4y ago
  Won't 3090 prices go down a good bit after the 4xxx series cards start coming out in a couple of months?
  - kajecounterhack 4y ago
    My guess is "not as much as most folks seem to think" Comparable 24gb vram 4xxx cards are also 1 slot bigger and many watts hungrier. If you want to be able to use a 800w PSU + 3 slots, and just need 24gb vram for say, running inference on a big diffusion model, then 3090 is still going to be your only option for a while. They're pretty well priced right now, at $1k. If you need one, not much reason to wait, your time is probably worth more than saving a couple hundred bucks.
    - KaoruAoiShiho 4y ago
      Is there no consumer option bigger than 24gb next gen?
      - InvaderFizz 4y ago
        It appears as though the 4090 will be 24GB, but that card may also be almost $2k. Used 3090s on eBay are $800 all day long. That price may drop a bit in the next week or so, but not much, as that 24GB of VRAM is the main draw for that over a 3080Ti.
      - tootyskooty 4y ago
        The 4090ti is rumoured to be 48GB [1], but who knows when that will release or how much it will cost. If you really need extra VRAM and don't mind longer inference, older used Tesla cards are an option. A used Tesla V100 32GB can be sometimes found on Ebay for 1500. [1] https://www.techpowerup.com/gpu-specs/geforce-rtx-4090-ti.c3917 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090-ti.c3...
        KaoruAoiShiho 4y ago
        Thank you great tip on V100.
papercrane 4y ago
You should get a Nvidia card with as much VRAM as possible. A 12 GB RTX 3060 is probably the most cost efficient at the moment. I don't think AMD is really viable for ML. Nvidia has the mind share in that segment, so nearly all tools will work with Nvidia, while very few support AMD.
- hedora 4y ago
  Not disagreeing, but my < $300 8GB AMD runs stable diffusion just fine.
  - capableweb 4y ago
    Guessing "just fine" is relative, but care to put a prompt + the settings you use, and how many it/s you get? I'm getting max ~8it/s on a 2080ti and I think even that feels slow sometimes so looking to upgrade my GPU now, curious to see what "just fine" means for you here.
    - hedora 4y ago
      100 seconds per batch of five (1.7it/s); default settings (512x512, etc) Performance seems to be prompt-independent; I'm using a docker container that seems to have disappeared, this modified version for amd cards with less than 10gb: https://github.com/basujindal/stable-diffusion https://github.com/basujindal/stable-diffusion and this workaround: https://github.com/RadeonOpenCompute/ROCm/issues/1756#issuecomment-1160386571 https://github.com/RadeonOpenCompute/ROCm/issues/1756#issuec... Anyway, don't "upgrade" to my budget AMD card. :-)
      - capableweb 4y ago
        > 100 seconds per batch of five (1.7it/s); default settings (512x512, etc) What settings precisely? Scale, steps and sampler being the most important ones (together with size, but you already shared that :) )
  - Uehreka 4y ago
cstejerean 4y ago
I picked up a 3090 Ti for this purpose given the price drop. The 24 GB of VRAM is hard to beat.
- ftufek 4y ago
  Not much difference between 3090 or 3090 ti though, while the price is usually 30% higher.
  - cstejerean 4y ago
    Mostly because the 3090 Ti was discounted to $1,000, at which point it's about the same as the non-Ti version.
  - wincy 4y ago
    The prices are identical now from what I’ve seen.
  - barrkel 4y ago
    The 3090 Ti also pumps out a lot more heat.
pbronez 4y ago
Lambda Labs crunched the numbers in Feb 2022 [0]. They concluded: “”” So, which GPUs to choose if you need an upgrade in early 2022 for Deep Learning? We feel there are two yes/no questions that help you choose between A100, A6000, and 3090. These three together probably cover most of the use cases in training Deep Learning models: Do you need multi-node distributed training? If the answer is yes, go for A100 80GB/40GB SXM4 because they are the only GPUs that support Infiniband. Without Infiniband, your distributed training simply would not scale. If the answer is no, see the next question. How big is your model? That helps you to choose between A100 PCIe (80GB), A6000 (48GB), and 3090 (24GB). A couple of 3090s are adequate for mainstream academic research. Choose A6000 if you work with a large image/language model and need multi-GPU training to scale efficiently. An A6000 system should cover most of the use cases in the context of a single node. Only choose A100 PCIe 80GB when working on extremely large models “”” [0] https://lambdalabs.com/blog/best-gpu-2022-sofar/ https://lambdalabs.com/blog/best-gpu-2022-sofar/
cavisne 4y ago
Needs a bit of an update but pretty good article https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni...

learndeeply 4y ago

This applies for all neural networks. Depending on how much money you're willing to spend, in descending order: DGX (computer with 8 A100s, $150,000), A100 (80GB, $15,000), A6000 ($5000), RTX 3090 ($1000).

moyix 4y ago
I just grabbed an A6000 for $3500 on eBay, so you can probably get a pretty decent deal on those now. They're pricey but IMO it's a great deal if you really need the VRAM (e.g. for training LLMs).
hedora 4y ago
I'd consider renting these, especially if you are just getting started: https://docs.paperspace.com/core/compute/machine-types/ https://docs.paperspace.com/core/compute/machine-types/ 1 x A100, 80GB is $3.19 / hour. 8 x A100, 80GB is about $25 / hour. They have much less expensive machines. I used to use them to run steam games, but now proton is just too darn good. Their low end machines are OK for CAD software that supports real-time raytracing.
- asciimike 4y ago
  Lots of good options for cheap GPU clouds, including Paperspace (mentioned above), Coreweave, and Crusoe Cloud (crusoecloud.com). Crusoe Cloud's angle is that our GPUs are powered off otherwise wasted energy and are carbon-reducing; running one for a year provides an emissions reduction equivalent to taking one car off the road. Disclosure, I'm head of product @ Crusoe Cloud. Feel free to ping me at mike at crusoecloud dot com if you've got questions or feedback.
  - pbronez 4y ago
    Cool idea. Tried to join your waitlist but the form throws errors. Could be my security measures, but I’m on mobile safari, toggled off all content blockers and still had an error.
    - asciimike 4y ago
      Odd, I just tested on Safari mobile and it went through without issues. Mind sending me an email at mike at crusoecloud dot com and we can get you set up?
  - douto 4y ago
    How can this be carbon-reducing? That sounds like a dubious marketing claim.
    - asciimike 4y ago
      TL;DR: we run data centers on-site at oil wells and take natural gas that would otherwise be flared (it can't be economically transported as natural gas in a pipeline or turned into electricity and transported) and combust it completely. Methane is a significantly more potent greenhouse gas than CO2, so it ends up being a net reduction in emissions vs what's currently happening. https://www.crusoeenergy.com/digital-flare-mitigation https://www.crusoeenergy.com/digital-flare-mitigation has some more information.
      - ip26 4y ago
        Is flare combustion more incomplete than your generation?
      - brimwats 4y ago
        on the other hand, does that mean you're obligated to be in oil producing countries? how does the geopolitics of that work?
  - hedora 4y ago
    How do you carbon offset? We have a bunch of diseased pine trees. Assuming we don't have enough time to cut them down and burn them in the next 12 months, and that I am unscrupulous, I could sell you carbon credits through one of the bigger exchanges. You would never know, and it would be completely above board. John Oliver had a particularly depressing segment on this recently. (Totally off topic, but I'd love to biochar them instead. Using them for lumber and shipping them offsite are non-starters. Any ideas, anyone?)
- lbotos 4y ago
  As someone who just fired off both paperspace boxes and coreweave to play around with stable diffusion, Paperspace limited me to some old janky hardware and never replied to my request to "unlock" the more modern cards. Coreweave let me get a modern card box first go round. I'd go with coreweave if you are a weekend hacker who just wants access to a lil bit of beefy GPU. YMMV.
- cwkoss 4y ago
  Are there "mining pools" for running AI jobs and getting paid in crypto?
  - knicholes 4y ago
    There's vast.ai, but you get paid in USD.
UncleOxidant 4y ago
Won't 3090 prices go down a good bit after the 4xxx series cards start coming out in a couple of months?
- kajecounterhack 4y ago
  My guess is "not as much as most folks seem to think" Comparable 24gb vram 4xxx cards are also 1 slot bigger and many watts hungrier. If you want to be able to use a 800w PSU + 3 slots, and just need 24gb vram for say, running inference on a big diffusion model, then 3090 is still going to be your only option for a while. They're pretty well priced right now, at $1k. If you need one, not much reason to wait, your time is probably worth more than saving a couple hundred bucks.
  - KaoruAoiShiho 4y ago
    Is there no consumer option bigger than 24gb next gen?
    - InvaderFizz 4y ago
      It appears as though the 4090 will be 24GB, but that card may also be almost $2k. Used 3090s on eBay are $800 all day long. That price may drop a bit in the next week or so, but not much, as that 24GB of VRAM is the main draw for that over a 3080Ti.
    - tootyskooty 4y ago
      The 4090ti is rumoured to be 48GB [1], but who knows when that will release or how much it will cost. If you really need extra VRAM and don't mind longer inference, older used Tesla cards are an option. A used Tesla V100 32GB can be sometimes found on Ebay for 1500. [1] https://www.techpowerup.com/gpu-specs/geforce-rtx-4090-ti.c3917 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090-ti.c3...
      - KaoruAoiShiho 4y ago
        Thank you great tip on V100.

papercrane 4y ago

You should get a Nvidia card with as much VRAM as possible. A 12 GB RTX 3060 is probably the most cost efficient at the moment. I don't think AMD is really viable for ML. Nvidia has the mind share in that segment, so nearly all tools will work with Nvidia, while very few support AMD.

hedora 4y ago
Not disagreeing, but my < $300 8GB AMD runs stable diffusion just fine.
- capableweb 4y ago
  Guessing "just fine" is relative, but care to put a prompt + the settings you use, and how many it/s you get? I'm getting max ~8it/s on a 2080ti and I think even that feels slow sometimes so looking to upgrade my GPU now, curious to see what "just fine" means for you here.
  - hedora 4y ago
    100 seconds per batch of five (1.7it/s); default settings (512x512, etc) Performance seems to be prompt-independent; I'm using a docker container that seems to have disappeared, this modified version for amd cards with less than 10gb: https://github.com/basujindal/stable-diffusion https://github.com/basujindal/stable-diffusion and this workaround: https://github.com/RadeonOpenCompute/ROCm/issues/1756#issuecomment-1160386571 https://github.com/RadeonOpenCompute/ROCm/issues/1756#issuec... Anyway, don't "upgrade" to my budget AMD card. :-)
    - capableweb 4y ago
      > 100 seconds per batch of five (1.7it/s); default settings (512x512, etc) What settings precisely? Scale, steps and sampler being the most important ones (together with size, but you already shared that :) )
- Uehreka 4y ago

cstejerean 4y ago

I picked up a 3090 Ti for this purpose given the price drop. The 24 GB of VRAM is hard to beat.

ftufek 4y ago
Not much difference between 3090 or 3090 ti though, while the price is usually 30% higher.
- cstejerean 4y ago
  Mostly because the 3090 Ti was discounted to $1,000, at which point it's about the same as the non-Ti version.
- wincy 4y ago
  The prices are identical now from what I’ve seen.
- barrkel 4y ago
  The 3090 Ti also pumps out a lot more heat.

pbronez 4y ago

Lambda Labs crunched the numbers in Feb 2022 [0]. They concluded: “”” So, which GPUs to choose if you need an upgrade in early 2022 for Deep Learning? We feel there are two yes/no questions that help you choose between A100, A6000, and 3090. These three together probably cover most of the use cases in training Deep Learning models: Do you need multi-node distributed training? If the answer is yes, go for A100 80GB/40GB SXM4 because they are the only GPUs that support Infiniband. Without Infiniband, your distributed training simply would not scale. If the answer is no, see the next question. How big is your model? That helps you to choose between A100 PCIe (80GB), A6000 (48GB), and 3090 (24GB). A couple of 3090s are adequate for mainstream academic research. Choose A6000 if you work with a large image/language model and need multi-GPU training to scale efficiently. An A6000 system should cover most of the use cases in the context of a single node. Only choose A100 PCIe 80GB when working on extremely large models “”” [0] https://lambdalabs.com/blog/best-gpu-2022-sofar/ https://lambdalabs.com/blog/best-gpu-2022-sofar/

cavisne 4y ago

Needs a bit of an update but pretty good article https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni...