3 ms·

Big Sur is twice as fast as our previous generation, which means we can train twice as fast and explore networks twice as large. Can a machine learning expert

by scott_s 11y ago

Big Sur is twice as fast as our previous generation, which means we can train twice as fast and explore networks twice as large. Can a machine learning expert comment on this claim? There are many domains in computing where doubling your compute power does not halve your execution time, or allow you to double the size of your input. (For example, your algorithms are worse than O(n), or you have hardware communication bottlenecks.)

willvarfar 11y ago
These are machine learning experts making the claim? Machine learning is usually embarrassingly parallel.
- scott_s 11y ago
  If I was reviewing a paper with bits on machine learning (presumably written by machine learning experts), I would ask machine learning experts I know to verify the claims I was not familiar with. I do know that a lot of ML algorithms become, at the low-level, matrix operations. Matrix operations are typically not embarrassingly parallel. Now, I can imagine a bunch of independent matrix operations, but whether you'd call the entire problem "embarrassingly parallel" would depend on how coarse the independent bits are.
  - hedgehog 11y ago
    In many cases you can train multiple instances of a neural net simultaneously in increments, average the weights together, and repeat. Making the underlying compute more efficient is an interesting problem.
spiantino 11y ago
Author here - in this case we are talking mostly about ~halving the time of a single minibatch in training a variety of standard neural networks. That type of training tends to be very hard to distribute. The servers also have 8 gpus in a single host, which is an additional performance boost and isn't included in the 2x above.
- hiddencost 11y ago
  I'm curious why this claim (distributed minibatch is hard) continues to be repeated. Am I misreading the state-of-the-art? There was a paper at Interspeech this year [0] where Amazon showed excellent scaling (54X speed up when distributed across 80 nodes) without loss of accuracy. [0] http://www.nikkostrom.com/publications/interspeech2015/strom_interspeech2015.pdf http://www.nikkostrom.com/publications/interspeech2015/strom...
chimtim 11y ago
This is not really true. ML applications in general do not scale linearly. There is the systems (scaling) overhead, and then there are the algorithm payoffs, which start to diminish depending on the algorithm. If they went from 4 to 8 GPUs, the only thing they can guarantee to double is how fast they burn power.

by scott_s 11y ago

willvarfar 11y ago

These are machine learning experts making the claim? Machine learning is usually embarrassingly parallel.

scott_s 11y ago
If I was reviewing a paper with bits on machine learning (presumably written by machine learning experts), I would ask machine learning experts I know to verify the claims I was not familiar with. I do know that a lot of ML algorithms become, at the low-level, matrix operations. Matrix operations are typically not embarrassingly parallel. Now, I can imagine a bunch of independent matrix operations, but whether you'd call the entire problem "embarrassingly parallel" would depend on how coarse the independent bits are.
- hedgehog 11y ago
  In many cases you can train multiple instances of a neural net simultaneously in increments, average the weights together, and repeat. Making the underlying compute more efficient is an interesting problem.

spiantino 11y ago

Author here - in this case we are talking mostly about ~halving the time of a single minibatch in training a variety of standard neural networks. That type of training tends to be very hard to distribute. The servers also have 8 gpus in a single host, which is an additional performance boost and isn't included in the 2x above.

hiddencost 11y ago
I'm curious why this claim (distributed minibatch is hard) continues to be repeated. Am I misreading the state-of-the-art? There was a paper at Interspeech this year [0] where Amazon showed excellent scaling (54X speed up when distributed across 80 nodes) without loss of accuracy. [0] http://www.nikkostrom.com/publications/interspeech2015/strom_interspeech2015.pdf http://www.nikkostrom.com/publications/interspeech2015/strom...

chimtim 11y ago

This is not really true. ML applications in general do not scale linearly. There is the systems (scaling) overhead, and then there are the algorithm payoffs, which start to diminish depending on the algorithm. If they went from 4 to 8 GPUs, the only thing they can guarantee to double is how fast they burn power.