2 ms·

I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude. RK'

I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude. RK's estimates are consistently best case estimates. But even if his estimates are wrong by a couple of decades, a lot can still happen before 2050.

jcranmer 8y ago
The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics. What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).
- trashtester 8y ago
  As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible. Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's. But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.
  - jcranmer 8y ago
    To be clear about one thing: When LINPACK measures FLOPS, it doesn't count how many instructions the computer executed. It uses a formula for how many floating-point operations it takes to solve an NxN linear equation and combines that with how long it took the computer to execute. So NVidia's tensor instructions actually contribute tens of operations per instruction retired. The prediction is attempting to model when a computer will exceed human computational power, which was estimated by Ray Kurzweil at 20PFLOPS, although it's unclear to me if we was specifically thinking of LINPACK Rmax or some other metric.

jcranmer 8y ago

The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics. What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).

trashtester 8y ago
As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible. Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's. But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.
- jcranmer 8y ago
  To be clear about one thing: When LINPACK measures FLOPS, it doesn't count how many instructions the computer executed. It uses a formula for how many floating-point operations it takes to solve an NxN linear equation and combines that with how long it took the computer to execute. So NVidia's tensor instructions actually contribute tens of operations per instruction retired. The prediction is attempting to model when a computer will exceed human computational power, which was estimated by Ray Kurzweil at 20PFLOPS, although it's unclear to me if we was specifically thinking of LINPACK Rmax or some other metric.