5 ms·

It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be

by rrss 7y ago

It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Great.

aey 7y ago
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
- anonuser123456 7y ago
  When Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.
  - deleted 7y ago
    [deleted]
  - rrss 7y ago
    Have semiconductor companies not been on the new architecture path already? It's not like they've just been doing die shrinks this whole time.
    - anonuser123456 7y ago
      Yes, but the emphasis now is much greater than before. The old rules of turn the scaling crank that defined the industry for decades are no longer helping.
- zak 7y ago
  Author of the blog post here. I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.
  - aey 7y ago
    What’s the perf per watt difference? It’s hard to compare perf per dollar. Your electricity costs and GPU costs maybe vastly different from mine.
- solidasparagus 7y ago
  For actual practitioners, TPUs are incredible. The cost/performance combo is unmatched. Now the real problem is, can you actually get a TPU pod in practice?
  - aey 7y ago
    My personal experience with crnns and lstms was that cutting training latency from 48 hours to 24 hours didn’t make a difference in our progress. Some architectures took 4x longer to run. 48 to 1 hour would have allowed us to run a few experiment per work day. That would be huge. If you told me that the next generation of TPUs could run the current 48 hour GPU training set in 1 hour, that would be something. But 2x faster won’t make a difference. Generally, startups for a new chip need to demonstrate 100x improvement over the general purpose approach to get funding. So this seems like another Google vanity project.
    - solidasparagus 7y ago
      I don't understand. Time-to-train is always a product of model and the amount of hardware you use. TPUs are fast and cheap allowing you to use more hardware for the same cost. The TPU design also makes it scale quite well. If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do that on GPUs. This is where the TPU shines since you can scale out without cost crippling you. I work on training on GPUs and the TPU is definitely an incredibly useful piece of hardware, not a vanity project.
      - aey 7y ago
        According to the post, it’s only 2x cheaper. So for the same dollars spent, I would cut my time from 48 hours to 24. But, TPUs are not standard, can’t be used for any other usecase. So those savings might not actually be realizable when everything is taken into account.
        p1esk 7y ago
m3at 7y ago
You might want to check the DAWN benchmark, it does just that: [1] https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-train-cost https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-t...
- rrss 7y ago
  Thanks, I've seen dawnbench, and it's definitely better than this presentation. Unfortunately, their cost numbers aren't actual cost, as they don't allow using AWS spot instance pricing (which is what fast.ai does). https://www.fast.ai/2018/04/30/dawnbench-fastai/ https://www.fast.ai/2018/04/30/dawnbench-fastai/
  - boulos 7y ago
    It was decided to disallow spot pricing and preemptible VMs because it’s super variable. On the day of a CVPR submission or something, you’re not going to pay the lowest rate (spot) nor perhaps be likely to get the VMs anyway (both spot and preemptible). Said another way, spot pricing results aren’t as reproducible in the research sense. It’s also a similar constant factor between the main providers. We felt that once you knew relative price/perf, you can choose to do whatever economic analysis you’d prefer (e.g., maybe if you don’t have any datacenter space yourself, there is no price you’d pay for hardware on-premises, or maybe you are willing to use spot or preemptible).
  - yaroslavvb 7y ago
    Divide by 3 to get pre-emptible price
boulos 7y ago
Disclosure: I work on Google Cloud. I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection. I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should I do inference on this chip from < startup >? I dunno, what’s it going to cost?”). Having to keep going back to some reference box is undesirable, but there’s nothing stopping someone from making a spreadsheet equivalent that translates these to dollars on the cloud providers (harder for on-prem submissions which reopens the “rent vs buy” debate).
- rrss 7y ago
  I'm not (currently) objecting to the actual MLPerf results. I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart. There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) works on TPUs, I would expect better.
  - zak 7y ago
    Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all. Our impression is that top-line performance comparisons independent of system size (like the comparison in the blog post) and performance-per-dollar comparisons are the easiest to understand, but we're certainly open to other ideas.
    - rrss 7y ago
      > top-line performance comparisons independent of system size Then do that, just don't do it all in one chart with the systems only specified in the fine print.
zak 7y ago
Author of the blog post here. Cloud TPUs are designed to maximize performance-per-dollar, so you are right that pure performance comparisons at maximum scale don't tell the whole story. The most straightforward performance-per-dollar comparisons would be among several different hardware configurations across the major public clouds. However, we haven't yet seen any other public cloud MLPerf submissions at scales comparable to Cloud TPU Pods, so there isn't currently a strong baseline available for comparison. It's also not clear whether public cloud networking will ultimately be able to match the performance of the network hardware that was used to produce the largest-scale on-premise MLPerf submissions.
- rrss 7y ago
  Why performance per dollar over performance per watt? Performance per watt doesn't care where the systems are (on-prem vs cloud) and should track perf/$ (unless perf/$ is mostly determined by subsidies).
  - frankchn 7y ago
    I think hardware acquisition costs can dominate GPU prices, so perf/watt might not be a realistic measure for the cost of running such large scale/high performance experiments. For instance, the DGX-2 has an MSRP of $399,000 and consumes 10 kW of power [1]. The average commercial electricity cost across the US is about $0.11/kWh [2], so a DGX-2 running at full tilt costs $1.10 an hour in electricity. Thus, a DGX-2 running at 100% utilization for 3 years costs $9,636 in electricity, which is ~2.5% of the cost of the box itself. Of course, you probably could get DGX-2s for a lot cheaper if you are buying 100 of them, but the acquisition costs are still going to be significant vis-a-vis power costs. [1]: https://www.anandtech.com/show/12587/nvidias-dgx2-sixteen-v100-gpus-30-tb-of-nvme-only-400k https://www.anandtech.com/show/12587/nvidias-dgx2-sixteen-v1... [2]: https://www.pacificpower.net/about/rr/cpc.html https://www.pacificpower.net/about/rr/cpc.html
  - zak 7y ago
    When comparing different hardware configurations within or between public clouds, power measurements generally aren't available, whereas prices or realistic price estimates generally are.

by rrss 7y ago

aey 7y ago

Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.

anonuser123456 7y ago
When Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.
- deleted 7y ago
  [deleted]
- rrss 7y ago
  Have semiconductor companies not been on the new architecture path already? It's not like they've just been doing die shrinks this whole time.
  - anonuser123456 7y ago
    Yes, but the emphasis now is much greater than before. The old rules of turn the scaling crank that defined the industry for decades are no longer helping.
zak 7y ago
Author of the blog post here. I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.
- aey 7y ago
  What’s the perf per watt difference? It’s hard to compare perf per dollar. Your electricity costs and GPU costs maybe vastly different from mine.
solidasparagus 7y ago
For actual practitioners, TPUs are incredible. The cost/performance combo is unmatched. Now the real problem is, can you actually get a TPU pod in practice?
- aey 7y ago
  My personal experience with crnns and lstms was that cutting training latency from 48 hours to 24 hours didn’t make a difference in our progress. Some architectures took 4x longer to run. 48 to 1 hour would have allowed us to run a few experiment per work day. That would be huge. If you told me that the next generation of TPUs could run the current 48 hour GPU training set in 1 hour, that would be something. But 2x faster won’t make a difference. Generally, startups for a new chip need to demonstrate 100x improvement over the general purpose approach to get funding. So this seems like another Google vanity project.
  - solidasparagus 7y ago
    I don't understand. Time-to-train is always a product of model and the amount of hardware you use. TPUs are fast and cheap allowing you to use more hardware for the same cost. The TPU design also makes it scale quite well. If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do that on GPUs. This is where the TPU shines since you can scale out without cost crippling you. I work on training on GPUs and the TPU is definitely an incredibly useful piece of hardware, not a vanity project.
    - aey 7y ago
      According to the post, it’s only 2x cheaper. So for the same dollars spent, I would cut my time from 48 hours to 24. But, TPUs are not standard, can’t be used for any other usecase. So those savings might not actually be realizable when everything is taken into account.
      - p1esk 7y ago

m3at 7y ago

You might want to check the DAWN benchmark, it does just that: [1] https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-train-cost https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-t...

rrss 7y ago
Thanks, I've seen dawnbench, and it's definitely better than this presentation. Unfortunately, their cost numbers aren't actual cost, as they don't allow using AWS spot instance pricing (which is what fast.ai does). https://www.fast.ai/2018/04/30/dawnbench-fastai/ https://www.fast.ai/2018/04/30/dawnbench-fastai/
- boulos 7y ago
  It was decided to disallow spot pricing and preemptible VMs because it’s super variable. On the day of a CVPR submission or something, you’re not going to pay the lowest rate (spot) nor perhaps be likely to get the VMs anyway (both spot and preemptible). Said another way, spot pricing results aren’t as reproducible in the research sense. It’s also a similar constant factor between the main providers. We felt that once you knew relative price/perf, you can choose to do whatever economic analysis you’d prefer (e.g., maybe if you don’t have any datacenter space yourself, there is no price you’d pay for hardware on-premises, or maybe you are willing to use spot or preemptible).
- yaroslavvb 7y ago
  Divide by 3 to get pre-emptible price

boulos 7y ago

Disclosure: I work on Google Cloud. I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection. I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should I do inference on this chip from < startup >? I dunno, what’s it going to cost?”). Having to keep going back to some reference box is undesirable, but there’s nothing stopping someone from making a spreadsheet equivalent that translates these to dollars on the cloud providers (harder for on-prem submissions which reopens the “rent vs buy” debate).

rrss 7y ago
I'm not (currently) objecting to the actual MLPerf results. I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart. There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) works on TPUs, I would expect better.
- zak 7y ago
  Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all. Our impression is that top-line performance comparisons independent of system size (like the comparison in the blog post) and performance-per-dollar comparisons are the easiest to understand, but we're certainly open to other ideas.
  - rrss 7y ago
    > top-line performance comparisons independent of system size Then do that, just don't do it all in one chart with the systems only specified in the fine print.

zak 7y ago

Author of the blog post here. Cloud TPUs are designed to maximize performance-per-dollar, so you are right that pure performance comparisons at maximum scale don't tell the whole story. The most straightforward performance-per-dollar comparisons would be among several different hardware configurations across the major public clouds. However, we haven't yet seen any other public cloud MLPerf submissions at scales comparable to Cloud TPU Pods, so there isn't currently a strong baseline available for comparison. It's also not clear whether public cloud networking will ultimately be able to match the performance of the network hardware that was used to produce the largest-scale on-premise MLPerf submissions.

rrss 7y ago
Why performance per dollar over performance per watt? Performance per watt doesn't care where the systems are (on-prem vs cloud) and should track perf/$ (unless perf/$ is mostly determined by subsidies).
- frankchn 7y ago
  I think hardware acquisition costs can dominate GPU prices, so perf/watt might not be a realistic measure for the cost of running such large scale/high performance experiments. For instance, the DGX-2 has an MSRP of $399,000 and consumes 10 kW of power [1]. The average commercial electricity cost across the US is about $0.11/kWh [2], so a DGX-2 running at full tilt costs $1.10 an hour in electricity. Thus, a DGX-2 running at 100% utilization for 3 years costs $9,636 in electricity, which is ~2.5% of the cost of the box itself. Of course, you probably could get DGX-2s for a lot cheaper if you are buying 100 of them, but the acquisition costs are still going to be significant vis-a-vis power costs. [1]: https://www.anandtech.com/show/12587/nvidias-dgx2-sixteen-v100-gpus-30-tb-of-nvme-only-400k https://www.anandtech.com/show/12587/nvidias-dgx2-sixteen-v1... [2]: https://www.pacificpower.net/about/rr/cpc.html https://www.pacificpower.net/about/rr/cpc.html
- zak 7y ago
  When comparing different hardware configurations within or between public clouds, power measurements generally aren't available, whereas prices or realistic price estimates generally are.