6 ms·

At this pricing level, with this amount of RAM, I suspect a lot of use cases will be with ML and GenAI. Benchmarks for these use cases would have been interesti

At this pricing level, with this amount of RAM, I suspect a lot of use cases will be with ML and GenAI. Benchmarks for these use cases would have been interesting.

janalsncm 3y ago
To be honest, the best benchmark you can run is your own training code. Everything else is a guess. When I tested the A6000 against the H100, there wasn’t that big of a boost from the newer card. Perhaps GPU operations weren’t the bottleneck in that case.
- RcouF1uZ4gsC 3y ago
  > To be honest, the best benchmark you can run is your own training code. Everything else is a guess. Yes, but the point of a review with benchmarks is that it is expensive and time-consuming for a customer to acquire the hardware just to run their own benchmarks on. Stable Diffusion and various LLMs are available pretty easily. A simple benchmark that this version of stable diffusion/llm was used with these settings and this is how long it took to produce image/we got this many tokens/sec would be a nice comparison that you are in a good position to do with access to all the hardware.
  - ex3ndr 3y ago
    This would end up memory-bound for LLMs (and you would better get x3 4090 for this) or by compute for diffusion models (and again 4090 is like 10% slower than H100 for 1% of the price).
  - janalsncm 3y ago
    I get what you’re saying, but I think too much complexity hides in those numbers. Some of the optimizations for LLMs (flash attention) are hardware specific. And anyways, it’s not that expensive. You can rent the same gpu from cloud providers for a few dollars per hour. If you are serious about buying a GPU it is an extremely small cost in terms of time and money compared to the price of the gpu itself. The alternative is trusting an online review running some training or inference code which is likely not comparable to what most people are doing.
    - michaelt 3y ago
      > You can rent the same gpu from cloud providers for a few dollars per hour. The big three cloud providers don't offer things like the RTX 5000 32GB or the 4090 do they? AWS will happily rent me a H100, A100, V100, K80, A10G, T4, or M60. Not to mention a Trainium, Inferentia, Inferentia2, Gaudi or Qualcomm AI 100. And don't forget to benchmark each one of those in a 1, 2, 4, and 8 GPU configuration; and with a variety of batch sizes; and with and without distributed training. Remember to work out the performance-per-dollar for on-demand, reserved, and spot instances, times three different cloud providers. Now, to compare to on-premise pricing we start by calling our air conditioning and backup generator vendors...
      - selfhoster11 3y ago
        I don't see why the parent comment would have meant the big 3 cloud companies. The GPU rental space is pretty commoditised, unlike for instance CDN/object storage where staying locked into a single vendor makes things simpler.
        michaelt 3y ago
        Not explicitly, but RcouF1uZ4gsC talked about "an extremely small cost in terms of time and money" Starting a VM with a cloud provider you already know and use may be extremely fast, but adopting a new cloud provider isn't. Even if your organisation is so unbureaucratic you can get a new provider set up without any due diligence work, every cloud provider comes with their own oddities. Will they have a wacky set of default firewall rules? Will they only offer Debian not Ubuntu, if you want an instance with the CUDA drivers already installed? Will they insist you learn what a 'provisioned iops' is before letting you start an instance? Will some GPUs only be available in certain regions? Will accounts have quota limits that vary by region, availability zone and GPU type, some of which default to zero?
        janalsncm 3y ago
        This isn’t migrating your entire app to a new ecosystem, this is starting up an empty machine, cloning a repo, opening Jupyter, and running some dummy training tasks with dummy training data. It takes 5 minutes from setup to teardown. It’s not difficult.
malkia 3y ago
Content creation tools (Maya, Blender, in-house) are another good candidate. Any game level editor would much apreciate extra GPU memory :)
ex3ndr 3y ago
it is 20-30% slower than 4090 or H100 in compute, the only improvement is slightly more RAM. This card is not for ML (on purpose) - it is for more enterprisish tasks: some advanced video streaming/rendering, virtualization, etc.
- michaelt 3y ago
  It's for some ML tasks. Just not large language models. If you're making, say, an ML-based on-premise CCTV system and you need to run several large ResNets at the same time? And you don't want to go rack-mounted, as some sites don't have a data centre? And you want the longer lifecycle and guaranteed spare parts availability of an enterprise product line? This could be the card for you. Admittedly it's a rip-off, but the Workstation/Quadro line always has been. Honestly I'm not sure how healthy the workstation market is right now - with the rise of work-from-home and hybrid working, I don't see many people using huge desktops any more. And when Adobe puts a powerful generative AI feature into Photoshop, they don't expect users to upgrade to powerful GPUs - they run it in the cloud, so it works for users with puny GPUs and Adobe can get that sweet sweet recurring revenue.
  - ex3ndr 3y ago
    No, this cards would be inferior to 4090, that you can get everywhere and you can buy 3 of them for "spare parts"
- YetAnotherNick 3y ago
  Comparison with 4090 is irrelevant as 4090 legally cannot legally be used for commercial usecase, and most companies just follow the law.
  - lreeves 3y ago
    Sorry what does that mean? There are many people creating content and software for commercial use cases with these cards and I've never seen a EULA anywhere preventing this.

At this pricing level, with this amount of RAM, I suspect a lot of use cases will be with ML and GenAI. Benchmarks for these use cases would have been interesting.

janalsncm 3y ago

To be honest, the best benchmark you can run is your own training code. Everything else is a guess. When I tested the A6000 against the H100, there wasn’t that big of a boost from the newer card. Perhaps GPU operations weren’t the bottleneck in that case.

RcouF1uZ4gsC 3y ago
> To be honest, the best benchmark you can run is your own training code. Everything else is a guess. Yes, but the point of a review with benchmarks is that it is expensive and time-consuming for a customer to acquire the hardware just to run their own benchmarks on. Stable Diffusion and various LLMs are available pretty easily. A simple benchmark that this version of stable diffusion/llm was used with these settings and this is how long it took to produce image/we got this many tokens/sec would be a nice comparison that you are in a good position to do with access to all the hardware.
- ex3ndr 3y ago
  This would end up memory-bound for LLMs (and you would better get x3 4090 for this) or by compute for diffusion models (and again 4090 is like 10% slower than H100 for 1% of the price).
- janalsncm 3y ago
  I get what you’re saying, but I think too much complexity hides in those numbers. Some of the optimizations for LLMs (flash attention) are hardware specific. And anyways, it’s not that expensive. You can rent the same gpu from cloud providers for a few dollars per hour. If you are serious about buying a GPU it is an extremely small cost in terms of time and money compared to the price of the gpu itself. The alternative is trusting an online review running some training or inference code which is likely not comparable to what most people are doing.
  - michaelt 3y ago
    > You can rent the same gpu from cloud providers for a few dollars per hour. The big three cloud providers don't offer things like the RTX 5000 32GB or the 4090 do they? AWS will happily rent me a H100, A100, V100, K80, A10G, T4, or M60. Not to mention a Trainium, Inferentia, Inferentia2, Gaudi or Qualcomm AI 100. And don't forget to benchmark each one of those in a 1, 2, 4, and 8 GPU configuration; and with a variety of batch sizes; and with and without distributed training. Remember to work out the performance-per-dollar for on-demand, reserved, and spot instances, times three different cloud providers. Now, to compare to on-premise pricing we start by calling our air conditioning and backup generator vendors...
    - selfhoster11 3y ago
      I don't see why the parent comment would have meant the big 3 cloud companies. The GPU rental space is pretty commoditised, unlike for instance CDN/object storage where staying locked into a single vendor makes things simpler.
      - michaelt 3y ago
        Not explicitly, but RcouF1uZ4gsC talked about "an extremely small cost in terms of time and money" Starting a VM with a cloud provider you already know and use may be extremely fast, but adopting a new cloud provider isn't. Even if your organisation is so unbureaucratic you can get a new provider set up without any due diligence work, every cloud provider comes with their own oddities. Will they have a wacky set of default firewall rules? Will they only offer Debian not Ubuntu, if you want an instance with the CUDA drivers already installed? Will they insist you learn what a 'provisioned iops' is before letting you start an instance? Will some GPUs only be available in certain regions? Will accounts have quota limits that vary by region, availability zone and GPU type, some of which default to zero?
        janalsncm 3y ago
        This isn’t migrating your entire app to a new ecosystem, this is starting up an empty machine, cloning a repo, opening Jupyter, and running some dummy training tasks with dummy training data. It takes 5 minutes from setup to teardown. It’s not difficult.

malkia 3y ago

Content creation tools (Maya, Blender, in-house) are another good candidate. Any game level editor would much apreciate extra GPU memory :)

ex3ndr 3y ago

it is 20-30% slower than 4090 or H100 in compute, the only improvement is slightly more RAM. This card is not for ML (on purpose) - it is for more enterprisish tasks: some advanced video streaming/rendering, virtualization, etc.

michaelt 3y ago
It's for some ML tasks. Just not large language models. If you're making, say, an ML-based on-premise CCTV system and you need to run several large ResNets at the same time? And you don't want to go rack-mounted, as some sites don't have a data centre? And you want the longer lifecycle and guaranteed spare parts availability of an enterprise product line? This could be the card for you. Admittedly it's a rip-off, but the Workstation/Quadro line always has been. Honestly I'm not sure how healthy the workstation market is right now - with the rise of work-from-home and hybrid working, I don't see many people using huge desktops any more. And when Adobe puts a powerful generative AI feature into Photoshop, they don't expect users to upgrade to powerful GPUs - they run it in the cloud, so it works for users with puny GPUs and Adobe can get that sweet sweet recurring revenue.
- ex3ndr 3y ago
  No, this cards would be inferior to 4090, that you can get everywhere and you can buy 3 of them for "spare parts"
YetAnotherNick 3y ago
Comparison with 4090 is irrelevant as 4090 legally cannot legally be used for commercial usecase, and most companies just follow the law.
- lreeves 3y ago
  Sorry what does that mean? There are many people creating content and software for commercial use cases with these cards and I've never seen a EULA anywhere preventing this.