5 ms·

In 2001, the fastest supercomputer in the world was ASCI White. It cost $110M, weighed 106 tons, consumed 3MW of power (plus 3MW for cooling), and had a peak sp

In 2001, the fastest supercomputer in the world was ASCI White. It cost $110M, weighed 106 tons, consumed 3MW of power (plus 3MW for cooling), and had a peak speed of 12.3 TFLOPS. Right now, sitting under my desk is a RTX 2080 Ti GPU which cost around $1000, weighs 3 pounds, draws a maximum of 250 watts, and has a peak speed of 13.4 TFLOPS [1]. We truly live in amazing times. [1] Not quite a fair comparison: the GPU is using 32-bit floating-point, while ASCI White used 64-bit. But for many applications, the precision difference doesn't matter.

nullc 7y ago
> while ASCI White used 64-bit. x87 is 80-bit. :)
- alkonaut 7y ago
  ASCI White wasn't Intel (it was IBM Power3)
  - nullc 7y ago
    oh, I was thinking ASCI red! fair enough!
sillysaurusx 7y ago
It's not fast enough. After having access to 160 TPUs, it's physically painful to use anything else. I hope in 20 years I'll have the equivalent of 160 TPUs under my desk. Hopefully less. The reason it's not fast enough is that ... there's so much you can do! People don't know. You can't really know until you have access to such a vast amount of horsepower, and can apply it to whatever you want. You might think "What could I possibly use it for?" but there are so many things. The most important thing you can use it for is fun, and intellectual gratification. You can train ML models just to see what they do. And as AI Dungeon shows, sometimes you win the lottery. I can't wait for the future. It's going to be so cool.
- fluffything 7y ago
  How do you program a TPU ? I looked to use some for solving PDEs, but Google had literally zero documentation on how to cross-compile C to TPUs, launch kernels, etc. AFAICT, you either use tensorflow or some other product that supports them, and for which the TPU code is not open source, or you can't use TPUs at all.
  - sillysaurusx 7y ago
    Do these help? https://github.com/google/jax/tree/master/cloud_tpu_colabs https://github.com/google/jax/tree/master/cloud_tpu_colabs The ODE solver might be close to what you want. I use Tensorflow 1.15. The world has been steadily pushing for Tensorflow 2.0 or Jax, but I like the simplicity of the Session model. It's so simple you can explain it in one sentence: it's an object that runs commands. Tell the session to connect to the TPU, and it will run all those commands on the TPU. Jax is new to me (and to everyone; they just released it). But it looks like Google is pouring some serious R&D into it. Two things help a lot. One, twitter. You can get a direct line to the people who actually make these beasts. Exploit it when you can. Like you, I dislike using a black box, and I'm intensely interested in the details of how to communicate with a TPU at a low level. I recently asked someone on the jax team about it here: https://twitter.com/theshawwn/status/1213221594052599808 https://twitter.com/theshawwn/status/1213221594052599808 Two, TFRC support has been incredibly helpful. https://www.tensorflow.org/tfrc https://www.tensorflow.org/tfrc I don't know who they have working the support channels, but those guys and gals are some of the most helpful and cheerful people I've come across. I often asked them very technical questions and to my surprise, they followed up with an A+ response almost every time, usually the next day. Pytorch is giving TF a real run for its money, and to be honest I once felt it was a mistake to invest so much time into Tensorflow. But it turned out to be a big advantage due to Google's investment in the overall ecosystem. TPUs are something that only Google has the resources to pull off. Note that the traditional path towards "just get a TPU up and running and start playing with it" is to use one of their Colab notebooks on the topic. https://cloud.google.com/tpu/docs/colabs https://cloud.google.com/tpu/docs/colabs I've been implicitly steering you away from these because you seem (like me) to want to know more of the low-level details. Those notebooks are designed to let ML researchers get results quickly, not for hardware enthusiasts to exploit heavy metal. The jax notebooks felt much more satisfying in that regard.
mlang23 7y ago
However, since you seem to count the GPU, fully utilising a modern system is definitely not easy.
- creato 7y ago
  It's not easy to fully (or even partially) utilize a supercomputer from 2001 either. Or a modern supercomputer for that matter.
mrb 7y ago
For a fair comparison (fp64 vs fp64): «A single GPU card like the AMD Radeon MI60 has more computing power than year 2000 supercomputer ASCI Red (fastest supercomputer in the TOP500 list of June 2000): • MI60: 7.4 TFLOPS (FP64) • ASCI Red: 3.2 TFLOPS (FP64) » https://mobile.twitter.com/zorinaq/status/1124912125187465216 https://mobile.twitter.com/zorinaq/status/112491212518746521...
m4rtink 7y ago
Also the supercomputer likely had a substantial ammount of solid state & fast spinning storage, even back then. Thats also often overlooked in these comparisons, not just the difference in precission.
- pixl97 7y ago
  A single 80mm nvme ssd would likely be faster than a significant amount of that supercomputers storage. In 2000 a million IOPS was a lofty target. Now we can do it on a single device.
  - dgacmu 7y ago
    ASCI red had 1TB of DRAM and 12TB of disk. Not bad, but three NVMe drives would clobber it. Putting that much DRAM in a box is still expensive today, but entirely feasible for about $5k-10k.
gruez 7y ago
Comparing GPU floating point performance with CPU floating point performance is comparing apples and oranges. GPUs may have higher raw FLOPS, but they have issues with workloads that aren't massively parallel or require branching.
- Symmetry 7y ago
  That's true when you're comparing a GPU to a single CPU. But when you're comparing a GPU to an entire supercomputer the requirement that the workload has massive parallelism to use all available resources is present in both.
  - gruez 7y ago
    It's a little more complicated for that. The main problem is that the way GPUs are designed, their execution units share the same instruction pointer[1]. That's not an issue if you're multiplying matricides, but it's an issue anytime you have branches. Therefore, even when your workload is massively parallel, it doesn't necessarily mean that a GPU cluster would perform nearly as well as a CPU cluster with the same amount of FLOPs. [1] https://en.wikipedia.org/wiki/Thread_block_(CUDA_programming) https://en.wikipedia.org/wiki/Thread_block_(CUDA_programming...

nullc 7y ago

> while ASCI White used 64-bit. x87 is 80-bit. :)

alkonaut 7y ago
ASCI White wasn't Intel (it was IBM Power3)
- nullc 7y ago
  oh, I was thinking ASCI red! fair enough!

sillysaurusx 7y ago

It's not fast enough. After having access to 160 TPUs, it's physically painful to use anything else. I hope in 20 years I'll have the equivalent of 160 TPUs under my desk. Hopefully less. The reason it's not fast enough is that ... there's so much you can do! People don't know. You can't really know until you have access to such a vast amount of horsepower, and can apply it to whatever you want. You might think "What could I possibly use it for?" but there are so many things. The most important thing you can use it for is fun, and intellectual gratification. You can train ML models just to see what they do. And as AI Dungeon shows, sometimes you win the lottery. I can't wait for the future. It's going to be so cool.

fluffything 7y ago
How do you program a TPU ? I looked to use some for solving PDEs, but Google had literally zero documentation on how to cross-compile C to TPUs, launch kernels, etc. AFAICT, you either use tensorflow or some other product that supports them, and for which the TPU code is not open source, or you can't use TPUs at all.
- sillysaurusx 7y ago
  Do these help? https://github.com/google/jax/tree/master/cloud_tpu_colabs https://github.com/google/jax/tree/master/cloud_tpu_colabs The ODE solver might be close to what you want. I use Tensorflow 1.15. The world has been steadily pushing for Tensorflow 2.0 or Jax, but I like the simplicity of the Session model. It's so simple you can explain it in one sentence: it's an object that runs commands. Tell the session to connect to the TPU, and it will run all those commands on the TPU. Jax is new to me (and to everyone; they just released it). But it looks like Google is pouring some serious R&D into it. Two things help a lot. One, twitter. You can get a direct line to the people who actually make these beasts. Exploit it when you can. Like you, I dislike using a black box, and I'm intensely interested in the details of how to communicate with a TPU at a low level. I recently asked someone on the jax team about it here: https://twitter.com/theshawwn/status/1213221594052599808 https://twitter.com/theshawwn/status/1213221594052599808 Two, TFRC support has been incredibly helpful. https://www.tensorflow.org/tfrc https://www.tensorflow.org/tfrc I don't know who they have working the support channels, but those guys and gals are some of the most helpful and cheerful people I've come across. I often asked them very technical questions and to my surprise, they followed up with an A+ response almost every time, usually the next day. Pytorch is giving TF a real run for its money, and to be honest I once felt it was a mistake to invest so much time into Tensorflow. But it turned out to be a big advantage due to Google's investment in the overall ecosystem. TPUs are something that only Google has the resources to pull off. Note that the traditional path towards "just get a TPU up and running and start playing with it" is to use one of their Colab notebooks on the topic. https://cloud.google.com/tpu/docs/colabs https://cloud.google.com/tpu/docs/colabs I've been implicitly steering you away from these because you seem (like me) to want to know more of the low-level details. Those notebooks are designed to let ML researchers get results quickly, not for hardware enthusiasts to exploit heavy metal. The jax notebooks felt much more satisfying in that regard.

mlang23 7y ago

However, since you seem to count the GPU, fully utilising a modern system is definitely not easy.

creato 7y ago
It's not easy to fully (or even partially) utilize a supercomputer from 2001 either. Or a modern supercomputer for that matter.

mrb 7y ago

For a fair comparison (fp64 vs fp64): «A single GPU card like the AMD Radeon MI60 has more computing power than year 2000 supercomputer ASCI Red (fastest supercomputer in the TOP500 list of June 2000): • MI60: 7.4 TFLOPS (FP64) • ASCI Red: 3.2 TFLOPS (FP64) » https://mobile.twitter.com/zorinaq/status/1124912125187465216 https://mobile.twitter.com/zorinaq/status/112491212518746521...

m4rtink 7y ago

Also the supercomputer likely had a substantial ammount of solid state & fast spinning storage, even back then. Thats also often overlooked in these comparisons, not just the difference in precission.

pixl97 7y ago
A single 80mm nvme ssd would likely be faster than a significant amount of that supercomputers storage. In 2000 a million IOPS was a lofty target. Now we can do it on a single device.
- dgacmu 7y ago
  ASCI red had 1TB of DRAM and 12TB of disk. Not bad, but three NVMe drives would clobber it. Putting that much DRAM in a box is still expensive today, but entirely feasible for about $5k-10k.

gruez 7y ago

Comparing GPU floating point performance with CPU floating point performance is comparing apples and oranges. GPUs may have higher raw FLOPS, but they have issues with workloads that aren't massively parallel or require branching.

Symmetry 7y ago
That's true when you're comparing a GPU to a single CPU. But when you're comparing a GPU to an entire supercomputer the requirement that the workload has massive parallelism to use all available resources is present in both.
- gruez 7y ago
  It's a little more complicated for that. The main problem is that the way GPUs are designed, their execution units share the same instruction pointer[1]. That's not an issue if you're multiplying matricides, but it's an issue anytime you have branches. Therefore, even when your workload is massively parallel, it doesn't necessarily mean that a GPU cluster would perform nearly as well as a CPU cluster with the same amount of FLOPs. [1] https://en.wikipedia.org/wiki/Thread_block_(CUDA_programming) https://en.wikipedia.org/wiki/Thread_block_(CUDA_programming...