4 ms·

Your citation is for 1k A100s, not 3.5k V100s. I think it's actually ~51 days on 3.5k V100s. Just to compare the GPUs, TF32 Tensor processing went from ~125 T

Your citation is for 1k A100s, not 3.5k V100s. I think it's actually ~51 days on 3.5k V100s. Just to compare the GPUs, TF32 Tensor processing went from ~125 TFlops to ~990. It then looks like they also dropped the precision to FP8, which gives you another 4x win. What's interesting is to look at how we're progressing in performance over time. In some sense, a bit slow? A V100 costs $10k at release; an H100 seems to be $40k. So we've only managed to halve the cost of a flop in 5 years. That seems.. much slower than what Moore's Law would have suggested.

winwang 3y ago
What about power draw? Quick google says V100 is max 300W while H100 is max 700W TDP, which makes the cost more favorable than 4x, so more like 7x less per flop. *assuming* the same electricity cost, which actually seems to have increased significantly. On a (minor) side note, it seems that $1.00 from 2018 is worth ~$1.20 in 2023. I wish more cost comparisons included inflation, because the past few years have had a lot of it.
- usaar333 3y ago
  Good call on power draw, but it seems dwarfed by capital cost. Assuming 3 year life (this tech gets outdated..), 300W constant consumption costs under $1k at $0.3/kwh. You need to be an order of magnitude higher in power usage before it really starts mattering. e.g. a Tesla3 consumes around 15 KW (20x the H100) while driving. That said it looks like flops/watt dropped by only 3.4x, which is also sub Moore's Law (3 years to halve power consumption)
  - TOMDM 3y ago
    I don't think flops/$ is enough to really capture the difference here. You couldn't replicate the scale of compute this allows no matter how many V100's you had. A huge amount of cost here is embodied in networking and memory. If one were to design a chip that cared only for flop/$ without caring for all of the interconnect and memory, then the 4090 is a much fairer comparison, and even then that card isn't designed for a flop/$ optimisation.
ac29 3y ago
> So we've only managed to halve the cost of a flop in 5 years. That seems.. much slower than what Moore's Law would have suggested. Moore's law says nothing about price or performance. H100 has about 4x as many transistors as V100, which is pretty close to what Moore's law would predict.
- esperent 3y ago
  > Moore's law says nothing about price or performance. Moore's law is about doubling of transistor count for the same price. At least that's always been my understanding. EDIT: I decided to look it up. Heres the original 1975 statement from him that led to the law: "The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years." So yup, it's about transistor density vs price.
  - jsjohnst 3y ago
    No, it’s not about price of a completely finished product in my humble opinion. It’s about the cost to produce the silicon component itself. Sure, it’s about money both ways, but the distinction is important. The V100 had up to 32gb ram, the H100 is 188gb. While there’s obviously transistors in ram, the counts being compared are for the GPU itself. I’d argue that a big chunk of the price difference in the V100 vs H100 is RAM.
    - usaar333 3y ago
      > I’d argue that a big chunk of the price difference in the V100 vs H100 is RAM. But that's the same Moore's Law problem when using the 'price' definition. 6x RAM 5 years later = 4x the cost? That's even slower than the FLOPS gains.
      - jsjohnst 3y ago
        Moore’s law never applied to ram and again, you aren’t comparing apples to apples either. There’s a lot of factors that go into building a complete system, so stop trying to apply an old cpu transistor adage to something it was never intended for and maybe you’ll have better success?

winwang 3y ago

What about power draw? Quick google says V100 is max 300W while H100 is max 700W TDP, which makes the cost more favorable than 4x, so more like 7x less per flop. *assuming* the same electricity cost, which actually seems to have increased significantly. On a (minor) side note, it seems that $1.00 from 2018 is worth ~$1.20 in 2023. I wish more cost comparisons included inflation, because the past few years have had a lot of it.

usaar333 3y ago
Good call on power draw, but it seems dwarfed by capital cost. Assuming 3 year life (this tech gets outdated..), 300W constant consumption costs under $1k at $0.3/kwh. You need to be an order of magnitude higher in power usage before it really starts mattering. e.g. a Tesla3 consumes around 15 KW (20x the H100) while driving. That said it looks like flops/watt dropped by only 3.4x, which is also sub Moore's Law (3 years to halve power consumption)
- TOMDM 3y ago
  I don't think flops/$ is enough to really capture the difference here. You couldn't replicate the scale of compute this allows no matter how many V100's you had. A huge amount of cost here is embodied in networking and memory. If one were to design a chip that cared only for flop/$ without caring for all of the interconnect and memory, then the 4090 is a much fairer comparison, and even then that card isn't designed for a flop/$ optimisation.

ac29 3y ago

> So we've only managed to halve the cost of a flop in 5 years. That seems.. much slower than what Moore's Law would have suggested. Moore's law says nothing about price or performance. H100 has about 4x as many transistors as V100, which is pretty close to what Moore's law would predict.

esperent 3y ago
> Moore's law says nothing about price or performance. Moore's law is about doubling of transistor count for the same price. At least that's always been my understanding. EDIT: I decided to look it up. Heres the original 1975 statement from him that led to the law: "The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years." So yup, it's about transistor density vs price.
- jsjohnst 3y ago
  No, it’s not about price of a completely finished product in my humble opinion. It’s about the cost to produce the silicon component itself. Sure, it’s about money both ways, but the distinction is important. The V100 had up to 32gb ram, the H100 is 188gb. While there’s obviously transistors in ram, the counts being compared are for the GPU itself. I’d argue that a big chunk of the price difference in the V100 vs H100 is RAM.
  - usaar333 3y ago
    > I’d argue that a big chunk of the price difference in the V100 vs H100 is RAM. But that's the same Moore's Law problem when using the 'price' definition. 6x RAM 5 years later = 4x the cost? That's even slower than the FLOPS gains.
    - jsjohnst 3y ago
      Moore’s law never applied to ram and again, you aren’t comparing apples to apples either. There’s a lot of factors that go into building a complete system, so stop trying to apply an old cpu transistor adage to something it was never intended for and maybe you’ll have better success?