3 ms·

> To be honest, the best benchmark you can run is your own training code. Everything else is a guess. Yes, but the point of a review with benchmarks is that it

> To be honest, the best benchmark you can run is your own training code. Everything else is a guess. Yes, but the point of a review with benchmarks is that it is expensive and time-consuming for a customer to acquire the hardware just to run their own benchmarks on. Stable Diffusion and various LLMs are available pretty easily. A simple benchmark that this version of stable diffusion/llm was used with these settings and this is how long it took to produce image/we got this many tokens/sec would be a nice comparison that you are in a good position to do with access to all the hardware.

ex3ndr 3y ago
This would end up memory-bound for LLMs (and you would better get x3 4090 for this) or by compute for diffusion models (and again 4090 is like 10% slower than H100 for 1% of the price).
janalsncm 3y ago
I get what you’re saying, but I think too much complexity hides in those numbers. Some of the optimizations for LLMs (flash attention) are hardware specific. And anyways, it’s not that expensive. You can rent the same gpu from cloud providers for a few dollars per hour. If you are serious about buying a GPU it is an extremely small cost in terms of time and money compared to the price of the gpu itself. The alternative is trusting an online review running some training or inference code which is likely not comparable to what most people are doing.
- michaelt 3y ago
  > You can rent the same gpu from cloud providers for a few dollars per hour. The big three cloud providers don't offer things like the RTX 5000 32GB or the 4090 do they? AWS will happily rent me a H100, A100, V100, K80, A10G, T4, or M60. Not to mention a Trainium, Inferentia, Inferentia2, Gaudi or Qualcomm AI 100. And don't forget to benchmark each one of those in a 1, 2, 4, and 8 GPU configuration; and with a variety of batch sizes; and with and without distributed training. Remember to work out the performance-per-dollar for on-demand, reserved, and spot instances, times three different cloud providers. Now, to compare to on-premise pricing we start by calling our air conditioning and backup generator vendors...
  - selfhoster11 3y ago
    I don't see why the parent comment would have meant the big 3 cloud companies. The GPU rental space is pretty commoditised, unlike for instance CDN/object storage where staying locked into a single vendor makes things simpler.
    - michaelt 3y ago
      Not explicitly, but RcouF1uZ4gsC talked about "an extremely small cost in terms of time and money" Starting a VM with a cloud provider you already know and use may be extremely fast, but adopting a new cloud provider isn't. Even if your organisation is so unbureaucratic you can get a new provider set up without any due diligence work, every cloud provider comes with their own oddities. Will they have a wacky set of default firewall rules? Will they only offer Debian not Ubuntu, if you want an instance with the CUDA drivers already installed? Will they insist you learn what a 'provisioned iops' is before letting you start an instance? Will some GPUs only be available in certain regions? Will accounts have quota limits that vary by region, availability zone and GPU type, some of which default to zero?
      - janalsncm 3y ago
        This isn’t migrating your entire app to a new ecosystem, this is starting up an empty machine, cloning a repo, opening Jupyter, and running some dummy training tasks with dummy training data. It takes 5 minutes from setup to teardown. It’s not difficult.
  - borissk 3y ago
    FinOps is a full time job - https://www.vitado.com/finops-how-to-gain-sponsorship-and-implement-a-new-structure/ https://www.vitado.com/finops-how-to-gain-sponsorship-and-im...
  - janalsncm 3y ago
    I don’t want to be promoting any particular cloud GPU services but you literally can rent those GPUs in single and multi GPU configurations if you need to. Even for H100 it’s only around $2/GPU/hour and comsumer grade ones like the 4090 are even cheaper of course.

ex3ndr 3y ago

This would end up memory-bound for LLMs (and you would better get x3 4090 for this) or by compute for diffusion models (and again 4090 is like 10% slower than H100 for 1% of the price).

janalsncm 3y ago

I get what you’re saying, but I think too much complexity hides in those numbers. Some of the optimizations for LLMs (flash attention) are hardware specific. And anyways, it’s not that expensive. You can rent the same gpu from cloud providers for a few dollars per hour. If you are serious about buying a GPU it is an extremely small cost in terms of time and money compared to the price of the gpu itself. The alternative is trusting an online review running some training or inference code which is likely not comparable to what most people are doing.

michaelt 3y ago
> You can rent the same gpu from cloud providers for a few dollars per hour. The big three cloud providers don't offer things like the RTX 5000 32GB or the 4090 do they? AWS will happily rent me a H100, A100, V100, K80, A10G, T4, or M60. Not to mention a Trainium, Inferentia, Inferentia2, Gaudi or Qualcomm AI 100. And don't forget to benchmark each one of those in a 1, 2, 4, and 8 GPU configuration; and with a variety of batch sizes; and with and without distributed training. Remember to work out the performance-per-dollar for on-demand, reserved, and spot instances, times three different cloud providers. Now, to compare to on-premise pricing we start by calling our air conditioning and backup generator vendors...
- selfhoster11 3y ago
  I don't see why the parent comment would have meant the big 3 cloud companies. The GPU rental space is pretty commoditised, unlike for instance CDN/object storage where staying locked into a single vendor makes things simpler.
  - michaelt 3y ago
    Not explicitly, but RcouF1uZ4gsC talked about "an extremely small cost in terms of time and money" Starting a VM with a cloud provider you already know and use may be extremely fast, but adopting a new cloud provider isn't. Even if your organisation is so unbureaucratic you can get a new provider set up without any due diligence work, every cloud provider comes with their own oddities. Will they have a wacky set of default firewall rules? Will they only offer Debian not Ubuntu, if you want an instance with the CUDA drivers already installed? Will they insist you learn what a 'provisioned iops' is before letting you start an instance? Will some GPUs only be available in certain regions? Will accounts have quota limits that vary by region, availability zone and GPU type, some of which default to zero?
    - janalsncm 3y ago
      This isn’t migrating your entire app to a new ecosystem, this is starting up an empty machine, cloning a repo, opening Jupyter, and running some dummy training tasks with dummy training data. It takes 5 minutes from setup to teardown. It’s not difficult.
- borissk 3y ago
  FinOps is a full time job - https://www.vitado.com/finops-how-to-gain-sponsorship-and-implement-a-new-structure/ https://www.vitado.com/finops-how-to-gain-sponsorship-and-im...
- janalsncm 3y ago
  I don’t want to be promoting any particular cloud GPU services but you literally can rent those GPUs in single and multi GPU configurations if you need to. Even for H100 it’s only around $2/GPU/hour and comsumer grade ones like the 4090 are even cheaper of course.