5 ms·

Is a high end computer anywhere near 20 quadrillion calculations per second? Edit: 20 quintillion would be 20 million teraflops. Sound right?

1024core 8y ago
Not that I'm much into "predictions", but for $4000 you can get a lot of GPU cards (assuming you can squeeze them all into a box, with the necessary power and connections).
- mrfusion 8y ago
  I was 1999 dollars so maybe 5 or 6k today? Do all the gpu cards add up to 20 quintillion?
  - gibybo 8y ago
    Typical $500 graphics card today will do around 10 teraflops. You'd need 2k of them ($1m) for 20 quadrillion flops (the prediction is quadrillion not quintillion).
- caymanjim 8y ago
  You can't really get a lot of GPU cards for $4000. Even two-year-old mid-range cards are $500/ea. The current high-end cards are $1500/ea.
ianai 8y ago
The last time I did the calculation we’re still lucky to get over a Tflop/s on a modern processor (if I did my math right). The 5 ghz wall and core counts contribute to the stagnation. The real hardware progress now is in non86 processors enabling higher core counts for the $, memory speed, and the eventual reality of permanent storage to be as fast as ram. Ie we’re likely to get “there” by increasing core counts and inter-process communication speeds.
mtmail 8y ago
Only supercomputers are in the range. https://en.wikipedia.org/wiki/TOP500#Top_500_ranking https://en.wikipedia.org/wiki/TOP500#Top_500_ranking Number 10 in the list for example can do 17 quadrillion calculations. (10^15 = petaFLOP (https://en.wikipedia.org/wiki/FLOPS https://en.wikipedia.org/wiki/FLOPS).
- pedrocr 8y ago
  This seems to match my napkin calculation of this prediction being off by around a factor of 10^5 (~100.000 processors being needed to match it). Although in the grand scheme of these predictions that's probably not too way off what it probably shows well is that the exponential curve Moore's law had given us is definitely not holding anymore. Since Kurzweil had used it or something similar to extrapolate his numbers this seems to bring into question the whole premise of the singularity prediction.
EdiX 8y ago
If I didn't make any mistakes 20 quadrillions are equivalent to 20000 TFLOPS. Wikipedia says you can get 12 TFLOPS with a high end graphics card for 1200$. 4000$ inflation adjusted is 6000$ which means you can get 5 of those for a total of 60TFLOPS which means he's off by a factor of 300. If you used CPUs instead (which is probably what he meant at the time) you can get 225 GOPS (integer operations, not floating point) for 374$ with a Core i7-9700K (source: wikipedia). With 6000$ you buy 16 of those for a total of 3.6 amazing TOPS, which makes Ray off by a factor of 5000.
- trashtester 8y ago
  I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude. RK's estimates are consistently best case estimates. But even if his estimates are wrong by a couple of decades, a lot can still happen before 2050.
  - jcranmer 8y ago
    The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics. What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).
    - trashtester 8y ago
      As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible. Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's. But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.
jcranmer 8y ago
If you interpret "calculation per second" as a FLOPS [as measured on LINPACK], 20 quadrillion is 20 PFLOPS. 20 PFLOPS puts you at #7 on TOP500. A looser interpretation is that you could count the number of integer adds you can push through a second multiplied by vector width. The CPU component of this would be in the region of 1.5 GHz (AVX-512 clock-speed) * 64 (8-bit adds/512-bit vector) * 2 issue ports * 10-20 cores (I don't know where the $4,000 in 1999 dollars hits in terms of pricepoint here [1]). You're looking at the region of low single-digit TOPS here. I don't know enough about the physical architecture of GPUs to estimate their theoretical headcount here, but I'd be surprised if you could push much past the teens of TOPS on those machines either. So, for the price point he predicted, we're off by a factor of 1,000. [1] I'm counting physical cores here. Hyperthreads don't add more actual ALUs, they just let two independent threads fill these up with operations.

1024core 8y ago

Not that I'm much into "predictions", but for $4000 you can get a lot of GPU cards (assuming you can squeeze them all into a box, with the necessary power and connections).

mrfusion 8y ago
I was 1999 dollars so maybe 5 or 6k today? Do all the gpu cards add up to 20 quintillion?
- gibybo 8y ago
  Typical $500 graphics card today will do around 10 teraflops. You'd need 2k of them ($1m) for 20 quadrillion flops (the prediction is quadrillion not quintillion).
caymanjim 8y ago
You can't really get a lot of GPU cards for $4000. Even two-year-old mid-range cards are $500/ea. The current high-end cards are $1500/ea.

ianai 8y ago

The last time I did the calculation we’re still lucky to get over a Tflop/s on a modern processor (if I did my math right). The 5 ghz wall and core counts contribute to the stagnation. The real hardware progress now is in non86 processors enabling higher core counts for the $, memory speed, and the eventual reality of permanent storage to be as fast as ram. Ie we’re likely to get “there” by increasing core counts and inter-process communication speeds.

mtmail 8y ago

Only supercomputers are in the range. https://en.wikipedia.org/wiki/TOP500#Top_500_ranking https://en.wikipedia.org/wiki/TOP500#Top_500_ranking Number 10 in the list for example can do 17 quadrillion calculations. (10^15 = petaFLOP (https://en.wikipedia.org/wiki/FLOPS https://en.wikipedia.org/wiki/FLOPS).

pedrocr 8y ago
This seems to match my napkin calculation of this prediction being off by around a factor of 10^5 (~100.000 processors being needed to match it). Although in the grand scheme of these predictions that's probably not too way off what it probably shows well is that the exponential curve Moore's law had given us is definitely not holding anymore. Since Kurzweil had used it or something similar to extrapolate his numbers this seems to bring into question the whole premise of the singularity prediction.

EdiX 8y ago

If I didn't make any mistakes 20 quadrillions are equivalent to 20000 TFLOPS. Wikipedia says you can get 12 TFLOPS with a high end graphics card for 1200$. 4000$ inflation adjusted is 6000$ which means you can get 5 of those for a total of 60TFLOPS which means he's off by a factor of 300. If you used CPUs instead (which is probably what he meant at the time) you can get 225 GOPS (integer operations, not floating point) for 374$ with a Core i7-9700K (source: wikipedia). With 6000$ you buy 16 of those for a total of 3.6 amazing TOPS, which makes Ray off by a factor of 5000.

trashtester 8y ago
I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude. RK's estimates are consistently best case estimates. But even if his estimates are wrong by a couple of decades, a lot can still happen before 2050.
- jcranmer 8y ago
  The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics. What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).
  - trashtester 8y ago
    As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible. Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's. But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.

jcranmer 8y ago

If you interpret "calculation per second" as a FLOPS [as measured on LINPACK], 20 quadrillion is 20 PFLOPS. 20 PFLOPS puts you at #7 on TOP500. A looser interpretation is that you could count the number of integer adds you can push through a second multiplied by vector width. The CPU component of this would be in the region of 1.5 GHz (AVX-512 clock-speed) * 64 (8-bit adds/512-bit vector) * 2 issue ports * 10-20 cores (I don't know where the $4,000 in 1999 dollars hits in terms of pricepoint here [1]). You're looking at the region of low single-digit TOPS here. I don't know enough about the physical architecture of GPUs to estimate their theoretical headcount here, but I'd be surprised if you could push much past the teens of TOPS on those machines either. So, for the price point he predicted, we're off by a factor of 1,000. [1] I'm counting physical cores here. Hyperthreads don't add more actual ALUs, they just let two independent threads fill these up with operations.