4 ms·

Okay, I have a question about one of his assertions here: > What may take a cluster to compute one year takes a consumer machine the next. Is that not partly

Okay, I have a question about one of his assertions here: > What may take a cluster to compute one year takes a consumer machine the next. Is that not partly because the hardware is ever improving? I realize this is a bit of exaggeration, but does not yesterday's cluster end up fitting onto the die of tomorrow's GPU? And then since it's all on a single die, is not the overhead of the interconnect drastically reduced? It takes less time to push information to the next core over when the interconnect is a couple micrometers of silicon instead of the couple meters of silicon, copper, and fiber needed when the next core is in the next rack over. Certainly improving the model will help; who hasn't marvelled at how better his code ran when he fixed that On^2 hot spot? But I can't help but think improving hardware plays a role too. Am I off base here?

yetihehe 8y ago
Off base not, but most of this "one year effect" is really improved algorithms, not better hardware. Hardware doesn't improve that fast (1000x in one year).
- heavenlyblue 8y ago
  One year effect? What exactly are they speaking about? The only reason deep learning exists is because by now we've finally learned how to build GPUs fast enough to run the algorithm invented in 1983. And let's be honest - most of the current, state-of-the-art algorithms only work today because they've got access to scaled up massive databases of data. You don't really need to be as smart here any more.
  - yetihehe 8y ago
    > One year effect? What exactly are they speaking about? From article: > What may take a cluster to compute one year takes a consumer machine the next. So, one year after introducing something you don't need cluster for computing, consumer machine will be sufficient. From first comment: > Is that not partly because the hardware is ever improving? [...] Certainly improving the model will help, who hasn't marvelled at how better his code ran when he fixed that On^2 hot spot. But I can't help but think improving hardware plays a role too. > Am I off base here? Now my comment was about original commenter being off base. Hardware improvement plays a role, but not during one year. 1000x speedup in a year is due to algorithms.
  - twtw 8y ago
    I don't mean to reduce the work of whichever researchers published in 1983 (are you thinking of Rumelhart et al 1986?), but gradient descent and reverse mode automatic differentiation are way older than 1983. Adaptive filters were being "trained" in the 50s using stochastic gradient descent.
    - _0ffh 8y ago
      And backpropagation (using the chain rule) was originally invented in the 60s in the field of control theory, IIRC.
      - twtw 8y ago
        > and reverse mode automatic differentiation
- MR4D 8y ago
  The author specifically states a hardware example: Neural Architecture Search: "32,400-43,200 GPU hours" Just over a year later: "single Nvidia GTX 1080Ti GPU, the search for architectures takes less than 16 hours" (1000x less) (paper)
- jfoutz 8y ago
  I think there was a paper about that in the 90’s. If your program required more than X years of supercomputer time, you will get your answer faster by waiting n years to start executing. If I remember correctly it was X=5 and n=2. Moore’s law was magical
heavenlyblue 8y ago
Yes, you are. Most of hard problems in computer science are that of highly-branched connectivity (both in terms of the number of connections that are mode from the compute node and in terms of the branching required in the algorithms). E.G. 1 - branched) if it takes you 100ms to send 1Gb of data over network, but you don't have to wait for it's result, then latency does not matter at all. E.G. 2 - number of connections to the node) if you have a matrix value that depends on a full row in a matrix, this matrix value somehow needs to be computationally connected to a full row in a matrix. Cluster or GPU, these physical connections need to be made either through Ethernet or by reading values from RAM. Again, the latency does not matter as long as you can pipeline these. So frankly, no. It's not the micrometers on the silicon that matters, it's just that our hardware had been scaled so much that a 50 year old compute cluster can be literally scaled down to a single die. It's _not_ more efficient, though.
- twtw 8y ago
  > be made either through Ethernet or by reading values from RAM Ok? This is not a rebuttal of the parent comment at all. You could also read and write the data on notecards and compute with a mechanical calculator, but that's totally irrelevant to the parent's point. Hardware matters a lot. Other things certainly matter as well, but saying the structure of the computation is fixed doesn't mean hardware can't change the performance (in absolute perf, perf/W, and perf/$) by many orders of magnitude.

yetihehe 8y ago

Off base not, but most of this "one year effect" is really improved algorithms, not better hardware. Hardware doesn't improve that fast (1000x in one year).

heavenlyblue 8y ago
One year effect? What exactly are they speaking about? The only reason deep learning exists is because by now we've finally learned how to build GPUs fast enough to run the algorithm invented in 1983. And let's be honest - most of the current, state-of-the-art algorithms only work today because they've got access to scaled up massive databases of data. You don't really need to be as smart here any more.
- yetihehe 8y ago
  > One year effect? What exactly are they speaking about? From article: > What may take a cluster to compute one year takes a consumer machine the next. So, one year after introducing something you don't need cluster for computing, consumer machine will be sufficient. From first comment: > Is that not partly because the hardware is ever improving? [...] Certainly improving the model will help, who hasn't marvelled at how better his code ran when he fixed that On^2 hot spot. But I can't help but think improving hardware plays a role too. > Am I off base here? Now my comment was about original commenter being off base. Hardware improvement plays a role, but not during one year. 1000x speedup in a year is due to algorithms.
- twtw 8y ago
  I don't mean to reduce the work of whichever researchers published in 1983 (are you thinking of Rumelhart et al 1986?), but gradient descent and reverse mode automatic differentiation are way older than 1983. Adaptive filters were being "trained" in the 50s using stochastic gradient descent.
  - _0ffh 8y ago
    And backpropagation (using the chain rule) was originally invented in the 60s in the field of control theory, IIRC.
    - twtw 8y ago
      > and reverse mode automatic differentiation
MR4D 8y ago
The author specifically states a hardware example: Neural Architecture Search: "32,400-43,200 GPU hours" Just over a year later: "single Nvidia GTX 1080Ti GPU, the search for architectures takes less than 16 hours" (1000x less) (paper)
jfoutz 8y ago
I think there was a paper about that in the 90’s. If your program required more than X years of supercomputer time, you will get your answer faster by waiting n years to start executing. If I remember correctly it was X=5 and n=2. Moore’s law was magical

heavenlyblue 8y ago

Yes, you are. Most of hard problems in computer science are that of highly-branched connectivity (both in terms of the number of connections that are mode from the compute node and in terms of the branching required in the algorithms). E.G. 1 - branched) if it takes you 100ms to send 1Gb of data over network, but you don't have to wait for it's result, then latency does not matter at all. E.G. 2 - number of connections to the node) if you have a matrix value that depends on a full row in a matrix, this matrix value somehow needs to be computationally connected to a full row in a matrix. Cluster or GPU, these physical connections need to be made either through Ethernet or by reading values from RAM. Again, the latency does not matter as long as you can pipeline these. So frankly, no. It's not the micrometers on the silicon that matters, it's just that our hardware had been scaled so much that a 50 year old compute cluster can be literally scaled down to a single die. It's _not_ more efficient, though.

twtw 8y ago
> be made either through Ethernet or by reading values from RAM Ok? This is not a rebuttal of the parent comment at all. You could also read and write the data on notecards and compute with a mechanical calculator, but that's totally irrelevant to the parent's point. Hardware matters a lot. Other things certainly matter as well, but saying the structure of the computation is fixed doesn't mean hardware can't change the performance (in absolute perf, perf/W, and perf/$) by many orders of magnitude.