5 ms·

Disclosure: I work on Google Cloud. I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets n

by boulos 7y ago

Disclosure: I work on Google Cloud. I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection. I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should I do inference on this chip from < startup >? I dunno, what’s it going to cost?”). Having to keep going back to some reference box is undesirable, but there’s nothing stopping someone from making a spreadsheet equivalent that translates these to dollars on the cloud providers (harder for on-prem submissions which reopens the “rent vs buy” debate).

rrss 7y ago
I'm not (currently) objecting to the actual MLPerf results. I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart. There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) works on TPUs, I would expect better.
- zak 7y ago
  Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all. Our impression is that top-line performance comparisons independent of system size (like the comparison in the blog post) and performance-per-dollar comparisons are the easiest to understand, but we're certainly open to other ideas.
  - rrss 7y ago
    > top-line performance comparisons independent of system size Then do that, just don't do it all in one chart with the systems only specified in the fine print.

by boulos 7y ago

rrss 7y ago

I'm not (currently) objecting to the actual MLPerf results. I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart. There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) works on TPUs, I would expect better.

zak 7y ago
Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all. Our impression is that top-line performance comparisons independent of system size (like the comparison in the blog post) and performance-per-dollar comparisons are the easiest to understand, but we're certainly open to other ideas.
- rrss 7y ago
  > top-line performance comparisons independent of system size Then do that, just don't do it all in one chart with the systems only specified in the fine print.