3 ms·

If I didn't make any mistakes 20 quadrillions are equivalent to 20000 TFLOPS. Wikipedia says you can get 12 TFLOPS with a high end graphics card for 1200$. 400

by EdiX 8y ago

If I didn't make any mistakes 20 quadrillions are equivalent to 20000 TFLOPS. Wikipedia says you can get 12 TFLOPS with a high end graphics card for 1200$. 4000$ inflation adjusted is 6000$ which means you can get 5 of those for a total of 60TFLOPS which means he's off by a factor of 300. If you used CPUs instead (which is probably what he meant at the time) you can get 225 GOPS (integer operations, not floating point) for 374$ with a Core i7-9700K (source: wikipedia). With 6000$ you buy 16 of those for a total of 3.6 amazing TOPS, which makes Ray off by a factor of 5000.

trashtester 8y ago
I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude. RK's estimates are consistently best case estimates. But even if his estimates are wrong by a couple of decades, a lot can still happen before 2050.
- jcranmer 8y ago
  The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics. What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).
  - trashtester 8y ago
    As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible. Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's. But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.
    - jcranmer 8y ago
      To be clear about one thing: When LINPACK measures FLOPS, it doesn't count how many instructions the computer executed. It uses a formula for how many floating-point operations it takes to solve an NxN linear equation and combines that with how long it took the computer to execute. So NVidia's tensor instructions actually contribute tens of operations per instruction retired. The prediction is attempting to model when a computer will exceed human computational power, which was estimated by Ray Kurzweil at 20PFLOPS, although it's unclear to me if we was specifically thinking of LINPACK Rmax or some other metric.

by EdiX 8y ago

trashtester 8y ago

I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude. RK's estimates are consistently best case estimates. But even if his estimates are wrong by a couple of decades, a lot can still happen before 2050.

jcranmer 8y ago
The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics. What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).
- trashtester 8y ago
  As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible. Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's. But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.
  - jcranmer 8y ago
    To be clear about one thing: When LINPACK measures FLOPS, it doesn't count how many instructions the computer executed. It uses a formula for how many floating-point operations it takes to solve an NxN linear equation and combines that with how long it took the computer to execute. So NVidia's tensor instructions actually contribute tens of operations per instruction retired. The prediction is attempting to model when a computer will exceed human computational power, which was estimated by Ray Kurzweil at 20PFLOPS, although it's unclear to me if we was specifically thinking of LINPACK Rmax or some other metric.