2 ms·
Bigger models don't (usually) mean dealing with higher quality images or something. They mean larger NN layers - and more of them - are able to fit in memory.
by nl 10y ago
Bigger models don't (usually) mean dealing with higher quality images or something. They mean larger NN layers - and more of them - are able to fit in memory.
This is important for image stuff, true, but it becomes really important for things like Memory Networks for NLP tasks and Neural Turing Machine-like architectures. Bigger networks mean they can "remember" further "backwards" in their dependencies, and so do things they can't do before.
That's GREAT news.
Also, the speed up is pretty significant. People train for weeks at the moment - a 10% speed up means often means savings days. This might not seem much, but it lets you iterate much, much quicker.
- emn13 10y agoFrom my own experience and from what I've seen others do, that's not what happens. People train for as long as they can afford to. If that is weeks, it's still going to be weeks. Instead, they're going to make the models slightly more sophisticated. If people really cared about 20% speedups that much, they could train for 20% fewer iterations at a slightly higher learning rate. Or they could use very slightly less deep networks. Or slightly less wide netwoks. Or slightly less connected networks. Or. most realistically, some combination of many very small changes. Of course it's nice to have faster machines, but this is hardly a big enough change to make a very noticeable difference, let alone a dramatic difference. There's nothing wrong with iterative improvement, but let's not pretend it's a revolution.