4 ms·

Off base not, but most of this "one year effect" is really improved algorithms, not better hardware. Hardware doesn't improve that fast (1000x in one year).

heavenlyblue 8y ago
One year effect? What exactly are they speaking about? The only reason deep learning exists is because by now we've finally learned how to build GPUs fast enough to run the algorithm invented in 1983. And let's be honest - most of the current, state-of-the-art algorithms only work today because they've got access to scaled up massive databases of data. You don't really need to be as smart here any more.
- yetihehe 8y ago
  > One year effect? What exactly are they speaking about? From article: > What may take a cluster to compute one year takes a consumer machine the next. So, one year after introducing something you don't need cluster for computing, consumer machine will be sufficient. From first comment: > Is that not partly because the hardware is ever improving? [...] Certainly improving the model will help, who hasn't marvelled at how better his code ran when he fixed that On^2 hot spot. But I can't help but think improving hardware plays a role too. > Am I off base here? Now my comment was about original commenter being off base. Hardware improvement plays a role, but not during one year. 1000x speedup in a year is due to algorithms.
- twtw 8y ago
  I don't mean to reduce the work of whichever researchers published in 1983 (are you thinking of Rumelhart et al 1986?), but gradient descent and reverse mode automatic differentiation are way older than 1983. Adaptive filters were being "trained" in the 50s using stochastic gradient descent.
  - _0ffh 8y ago
    And backpropagation (using the chain rule) was originally invented in the 60s in the field of control theory, IIRC.
    - twtw 8y ago
      > and reverse mode automatic differentiation
MR4D 8y ago
The author specifically states a hardware example: Neural Architecture Search: "32,400-43,200 GPU hours" Just over a year later: "single Nvidia GTX 1080Ti GPU, the search for architectures takes less than 16 hours" (1000x less) (paper)
jfoutz 8y ago
I think there was a paper about that in the 90’s. If your program required more than X years of supercomputer time, you will get your answer faster by waiting n years to start executing. If I remember correctly it was X=5 and n=2. Moore’s law was magical

heavenlyblue 8y ago

One year effect? What exactly are they speaking about? The only reason deep learning exists is because by now we've finally learned how to build GPUs fast enough to run the algorithm invented in 1983. And let's be honest - most of the current, state-of-the-art algorithms only work today because they've got access to scaled up massive databases of data. You don't really need to be as smart here any more.

yetihehe 8y ago
> One year effect? What exactly are they speaking about? From article: > What may take a cluster to compute one year takes a consumer machine the next. So, one year after introducing something you don't need cluster for computing, consumer machine will be sufficient. From first comment: > Is that not partly because the hardware is ever improving? [...] Certainly improving the model will help, who hasn't marvelled at how better his code ran when he fixed that On^2 hot spot. But I can't help but think improving hardware plays a role too. > Am I off base here? Now my comment was about original commenter being off base. Hardware improvement plays a role, but not during one year. 1000x speedup in a year is due to algorithms.
twtw 8y ago
I don't mean to reduce the work of whichever researchers published in 1983 (are you thinking of Rumelhart et al 1986?), but gradient descent and reverse mode automatic differentiation are way older than 1983. Adaptive filters were being "trained" in the 50s using stochastic gradient descent.
- _0ffh 8y ago
  And backpropagation (using the chain rule) was originally invented in the 60s in the field of control theory, IIRC.
  - twtw 8y ago
    > and reverse mode automatic differentiation

MR4D 8y ago

The author specifically states a hardware example: Neural Architecture Search: "32,400-43,200 GPU hours" Just over a year later: "single Nvidia GTX 1080Ti GPU, the search for architectures takes less than 16 hours" (1000x less) (paper)

jfoutz 8y ago

I think there was a paper about that in the 90’s. If your program required more than X years of supercomputer time, you will get your answer faster by waiting n years to start executing. If I remember correctly it was X=5 and n=2. Moore’s law was magical