From my own experience and from what I've seen others do, that's not what happens. People train for as long as they can afford to. If that is weeks, it's stil

From my own experience and from what I've seen others do, that's not what happens. People train for as long as they can afford to. If that is weeks, it's still going to be weeks. Instead, they're going to make the models slightly more sophisticated. If people really cared about 20% speedups that much, they could train for 20% fewer iterations at a slightly higher learning rate. Or they could use very slightly less deep networks. Or slightly less wide netwoks. Or slightly less connected networks. Or. most realistically, some combination of many very small changes. Of course it's nice to have faster machines, but this is hardly a big enough change to make a very noticeable difference, let alone a dramatic difference. There's nothing wrong with iterative improvement, but let's not pretend it's a revolution.