3 ms·

TPUs are perfectly positioned to capture this market. There are an endless number of reasons why, but to keep it short: Wanna see a magic trick? https://twitter

TPUs are perfectly positioned to capture this market. There are an endless number of reasons why, but to keep it short: Wanna see a magic trick? https://twitter.com/theshawwn/status/1286426454171975680 https://twitter.com/theshawwn/status/1286426454171975680 GPT-2 117M training at 1 million tokens/sec. Now, I don't have experience with DGX clusters, so I'm not going to make a firm statement. What I will say is that I, as an outsider, managed to achieve a performance level that is ~unheard of for GPT-2 training. And you can too; TPUs are pervasive. A TPUv2-512 isn't even as far as the gas pedal goes, either. v3-512 can train all of imagenet to 75.9% accuracy in 4 minutes: https://twitter.com/theshawwn/status/1223395022814339073 https://twitter.com/theshawwn/status/1223395022814339073 v3-1024 can do it in 2 minutes: https://twitter.com/theshawwn/status/1234654848114520065 https://twitter.com/theshawwn/status/1234654848114520065 I once attached a debugger to a training run during startup, after the infeed loop began (meaning it was feeding inputs to the TPU, but no training was happening yet; it was "winding up") and was shocked to discover that when I hit c to continue, it trained on all of imagenet in like 54 seconds. That blows the lid off of every perf result here (under "image classification"): https://mlperf.org/training-results-0-6 https://mlperf.org/training-results-0-6 (It's not a fair comparison, but it was quite astonishing to see the raw horsepower in action.) So, nVidia has some catching up to do. And I don't know if they'll be able to. The TPU ecosystem may be clunky at the moment, but boy is it effective. Your options are to invest your time in this ecosystem, which will likely be around in ten years, or in DGX-cluster-type knowledge, which ... might be less pervasive in 10 years. The distinguishing feature of a TPU is that it has a CPU on board. In fact, it has a CPU with 300GB of memory for every 8 cores. Friggin' love these things.

saberience 6y ago
1 milion per second of anything isn't very fast. Unless you're running that on a mobile phone.
xxxtentachyon 6y ago
Last time I worked on TPUs, a lot of very pervasive LA operations (e.g. Cholesky decomposition) were unoptimized and slow compared to the NN-style operations. I’m sure that can be fixed, but for the time being, it seems inappropriate for anything beyond the obvious NN operations.
rrss 6y ago
If "this market" means supercomputers (as 'Cthulhu_ was talking about), then TPUs are certainly not perfectly positioned to capture this market, since they are only capable of doing bfloat16 arithmetic with high performance, and AFAIK incapable of fp64 at all. That means a HPL score of zero. TPUs will probably dominate AI training, but not so much supercomputing.

saberience 6y ago

1 milion per second of anything isn't very fast. Unless you're running that on a mobile phone.

xxxtentachyon 6y ago

Last time I worked on TPUs, a lot of very pervasive LA operations (e.g. Cholesky decomposition) were unoptimized and slow compared to the NN-style operations. I’m sure that can be fixed, but for the time being, it seems inappropriate for anything beyond the obvious NN operations.

rrss 6y ago

If "this market" means supercomputers (as 'Cthulhu_ was talking about), then TPUs are certainly not perfectly positioned to capture this market, since they are only capable of doing bfloat16 arithmetic with high performance, and AFAIK incapable of fp64 at all. That means a HPL score of zero. TPUs will probably dominate AI training, but not so much supercomputing.