4 ms·

Well... this means you can train models that are very slightly larger in the same time. It wont scale linearly. And those slightly larger models will typically

by emn13 10y ago

Well... this means you can train models that are very slightly larger in the same time. It wont scale linearly. And those slightly larger models will typically be a very, very, very tiny bit better - quality scales far below linearly. Calling this great news is exaggerating greatly.

mrmondo 10y ago
also some times exaggerating == marketing
cs702 10y ago
Going by the specs, I would expect that training current models on the new Titan X will be be 1.3x to 2.0x faster than with the older Titan X. In practice, this means deep neural nets that now take four weeks to train will take two or three weeks instead. That's great news.
- Retric 10y ago
  If that maters, use slightly smaller networks.
  - astrodust 10y ago
    With that attitude you could fit your model on a 6502 with 16KB of memory. Give it up.
nl 10y ago
Bigger models don't (usually) mean dealing with higher quality images or something. They mean larger NN layers - and more of them - are able to fit in memory. This is important for image stuff, true, but it becomes really important for things like Memory Networks for NLP tasks and Neural Turing Machine-like architectures. Bigger networks mean they can "remember" further "backwards" in their dependencies, and so do things they can't do before. That's GREAT news. Also, the speed up is pretty significant. People train for weeks at the moment - a 10% speed up means often means savings days. This might not seem much, but it lets you iterate much, much quicker.
- emn13 10y ago
  From my own experience and from what I've seen others do, that's not what happens. People train for as long as they can afford to. If that is weeks, it's still going to be weeks. Instead, they're going to make the models slightly more sophisticated. If people really cared about 20% speedups that much, they could train for 20% fewer iterations at a slightly higher learning rate. Or they could use very slightly less deep networks. Or slightly less wide netwoks. Or slightly less connected networks. Or. most realistically, some combination of many very small changes. Of course it's nice to have faster machines, but this is hardly a big enough change to make a very noticeable difference, let alone a dramatic difference. There's nothing wrong with iterative improvement, but let's not pretend it's a revolution.

by emn13 10y ago

mrmondo 10y ago

also some times exaggerating == marketing

cs702 10y ago

Going by the specs, I would expect that training current models on the new Titan X will be be 1.3x to 2.0x faster than with the older Titan X. In practice, this means deep neural nets that now take four weeks to train will take two or three weeks instead. That's great news.

Retric 10y ago
If that maters, use slightly smaller networks.
- astrodust 10y ago
  With that attitude you could fit your model on a 6502 with 16KB of memory. Give it up.

nl 10y ago

Bigger models don't (usually) mean dealing with higher quality images or something. They mean larger NN layers - and more of them - are able to fit in memory. This is important for image stuff, true, but it becomes really important for things like Memory Networks for NLP tasks and Neural Turing Machine-like architectures. Bigger networks mean they can "remember" further "backwards" in their dependencies, and so do things they can't do before. That's GREAT news. Also, the speed up is pretty significant. People train for weeks at the moment - a 10% speed up means often means savings days. This might not seem much, but it lets you iterate much, much quicker.

emn13 10y ago
From my own experience and from what I've seen others do, that's not what happens. People train for as long as they can afford to. If that is weeks, it's still going to be weeks. Instead, they're going to make the models slightly more sophisticated. If people really cared about 20% speedups that much, they could train for 20% fewer iterations at a slightly higher learning rate. Or they could use very slightly less deep networks. Or slightly less wide netwoks. Or slightly less connected networks. Or. most realistically, some combination of many very small changes. Of course it's nice to have faster machines, but this is hardly a big enough change to make a very noticeable difference, let alone a dramatic difference. There's nothing wrong with iterative improvement, but let's not pretend it's a revolution.