8 ms·

And Nvidia's GPUs now include the same type of hardware that TPUs have, so there's no reason to believe that TPUs will win out over GPUs.

sillysaurusx 6y ago
The key difference between a TPU and a GPU is that a TPU has a CPU. It's an entire computer, not just a piece of hardware. Is nVidia moving in that direction?
- newsclues 6y ago
  They just bought ARM for 40$billion. I think they want to integrate CPU,GPU and high speed networks
- dikei 6y ago
  In term of cutting edge tech, They have their own GPUs, CPUs from ARM, Networking from Mellanox, so I'd say they're pretty much set to build a kick ass TPU.
- shaklee3 6y ago
  A TPU is a chip you cannot program. It's purpose built and can't run the fraction of the type of workloads that a GPU can.
  - sillysaurusx 6y ago
    I don't know where all of this misinformation is coming from or why, but, as someone who has spent the last year programming TPUs to do all kinds of things that a GPU can't do, this isn't true. Are we going to simply say "Nu uh" at each other, or do you want to throw down some specific examples so I can show you how mistaken they are?
    - shaklee3 6y ago
      Please show me the API where I can write a generic function on a TPU. I'm talking about writing something like a custom reduction or a peak search, not offloading a tensor flow model. I'll make it easier for you, directly from Google's website: TPUs Cloud TPUs are optimized for specific workloads. In some situations, you might want to use GPUs or CPUs on Compute Engine instances to run your machine learning workloads. Please tell me a workload a gpu can't do that a TPU can.
      - sillysaurusx 6y ago
        Sure, here you go: https://www.tensorflow.org/api_docs/python/tf/raw_ops https://www.tensorflow.org/api_docs/python/tf/raw_ops In my experience, well over 80% of these operations are implemented on TPU CPUs, and at least 60% are implemented on TPU cores. Again, if you give a specific example, I can simply write a program demonstrating that it works. What kind of custom reduction do you want? What's a peak search? As for workloads that GPUs can't do, we regularly train GANs at 500+ examples/sec across a total dataset size of >3M photos. Rather hard to do that with GPUs.
        shaklee3 6y ago
        Well, there you go. For one TensorFlow is not a generic framework like cuda is, so you lose a whole bunch of the configurability you have with cuda. So, for example, even though there is an FFT raw function, there doesn't appear to be a way to do more complicated FFTs, such as an overlap-save. This is trivial to do on a GPU, and is built into the library. The raw functions it provides is not direct access to the hardware and memory subsystem. It's a set of raw functions that is a small subset of the total problem space. And certainly if you are saying that running something on a TPU's CPU cores are in any way going to compete with a gpu, then I don't know what to tell you. You did not give an example of something GPUs can't do. all you said was that TPUs are faster for a specific function in your case.
        sillysaurusx 6y ago
        For one TensorFlow is not a generic framework like cuda is, so you lose a whole bunch of the configurability you have with cuda Why make generalizations like this? It's not true, and we've devolved back into the "nu uh" we originally started with. This is trivial to do on a GPU, and is built into the library Yes, I'm sure there are hardwired operations that are trivial to do on GPUs. That's not exactly a +1 in favor of generic programmability. There are also operations that are trivial to do on TPUs, such as CrossReplicaSum across a massive cluster of cores, or the various special-case Adam operations. This doesn't seem related to the claim that TPUs are less flexible. The raw functions it provides is not direct access to the hardware and memory subsystem. Not true. https://www.tensorflow.org/api_docs/python/tf/raw_ops/InplaceUpdate https://www.tensorflow.org/api_docs/python/tf/raw_ops/Inplac... Jax is also going to be giving even lower-level access than TF, which may interest you. You did not give an example of something GPUs can't do. all you said was that TPUs are faster for a specific function in your case. Well yeah, I care about achieving goals in my specific case, as you do yours. And simply getting together a VM that can feed 500 examples/sec to a set of GPUs is a massive undertaking in and of itself. TPUs make it more or less "easy" in comparison. (I won't say effortless, since it does take some effort to get yourself into the TPU programming mindset.)
    - deleted 6y ago
      [deleted]
    - sorenbouma 6y ago
      I'm a TPU user and I'd be interested to see a specific example of something that can be done on TPU but not GPU. Perhaps I'm just not experienced enough with the programming model, but I've found them to be strictly less flexible/more tricky than GPUs, especially for things like conditional execution, multiple graphs, variable size inputs and custom ops.
      - sillysaurusx 6y ago
        Sure! I'd love to chat TPUs. There's a #tpu discord channel on the MLPerf discord: https://github.com/shawwn/tpunicorn#ml-community https://github.com/shawwn/tpunicorn#ml-community The central reason that TPUs feel less flexible is Google's awful mistake in encouraging everyone to use TPUEstimator as the One True API For Doing TPU Programming. Getting off that API was the single biggest boost to my TPU skills. You can see an example of how to do that here: https://github.com/shawwn/ml-notes/blob/master/train_runner.py https://github.com/shawwn/ml-notes/blob/master/train_runner.... This is a repo that can train GPT-2 1.5B at 10 examples/sec on a TPUv3-8 (aka around 10k tokens/sec). Happy to answer any specific questions or peek at codebases you're hoping to run on TPUs.
        slaymaker1907 6y ago
        That doesn't answer the question of what a TPU can do that a GPU can't. I think the OP means impossible for the GPU, not just slower.
    - slaymaker1907 6y ago
      You can run basically any C program on a CUDA core even those requiring malloc. It may not be efficient but you can do it. Google themselves call GPUs general purpose and TPUs domain specific. https://cloud.google.com/blog/products/ai-machine-learning/what-makes-tpus-fine-tuned-for-deep-learning https://cloud.google.com/blog/products/ai-machine-learning/w...

sillysaurusx 6y ago

The key difference between a TPU and a GPU is that a TPU has a CPU. It's an entire computer, not just a piece of hardware. Is nVidia moving in that direction?

newsclues 6y ago
They just bought ARM for 40$billion. I think they want to integrate CPU,GPU and high speed networks
dikei 6y ago
In term of cutting edge tech, They have their own GPUs, CPUs from ARM, Networking from Mellanox, so I'd say they're pretty much set to build a kick ass TPU.
shaklee3 6y ago
A TPU is a chip you cannot program. It's purpose built and can't run the fraction of the type of workloads that a GPU can.
- sillysaurusx 6y ago
  I don't know where all of this misinformation is coming from or why, but, as someone who has spent the last year programming TPUs to do all kinds of things that a GPU can't do, this isn't true. Are we going to simply say "Nu uh" at each other, or do you want to throw down some specific examples so I can show you how mistaken they are?
  - shaklee3 6y ago
    Please show me the API where I can write a generic function on a TPU. I'm talking about writing something like a custom reduction or a peak search, not offloading a tensor flow model. I'll make it easier for you, directly from Google's website: TPUs Cloud TPUs are optimized for specific workloads. In some situations, you might want to use GPUs or CPUs on Compute Engine instances to run your machine learning workloads. Please tell me a workload a gpu can't do that a TPU can.
    - sillysaurusx 6y ago
      Sure, here you go: https://www.tensorflow.org/api_docs/python/tf/raw_ops https://www.tensorflow.org/api_docs/python/tf/raw_ops In my experience, well over 80% of these operations are implemented on TPU CPUs, and at least 60% are implemented on TPU cores. Again, if you give a specific example, I can simply write a program demonstrating that it works. What kind of custom reduction do you want? What's a peak search? As for workloads that GPUs can't do, we regularly train GANs at 500+ examples/sec across a total dataset size of >3M photos. Rather hard to do that with GPUs.
      - shaklee3 6y ago
        Well, there you go. For one TensorFlow is not a generic framework like cuda is, so you lose a whole bunch of the configurability you have with cuda. So, for example, even though there is an FFT raw function, there doesn't appear to be a way to do more complicated FFTs, such as an overlap-save. This is trivial to do on a GPU, and is built into the library. The raw functions it provides is not direct access to the hardware and memory subsystem. It's a set of raw functions that is a small subset of the total problem space. And certainly if you are saying that running something on a TPU's CPU cores are in any way going to compete with a gpu, then I don't know what to tell you. You did not give an example of something GPUs can't do. all you said was that TPUs are faster for a specific function in your case.
        sillysaurusx 6y ago
        For one TensorFlow is not a generic framework like cuda is, so you lose a whole bunch of the configurability you have with cuda Why make generalizations like this? It's not true, and we've devolved back into the "nu uh" we originally started with. This is trivial to do on a GPU, and is built into the library Yes, I'm sure there are hardwired operations that are trivial to do on GPUs. That's not exactly a +1 in favor of generic programmability. There are also operations that are trivial to do on TPUs, such as CrossReplicaSum across a massive cluster of cores, or the various special-case Adam operations. This doesn't seem related to the claim that TPUs are less flexible. The raw functions it provides is not direct access to the hardware and memory subsystem. Not true. https://www.tensorflow.org/api_docs/python/tf/raw_ops/InplaceUpdate https://www.tensorflow.org/api_docs/python/tf/raw_ops/Inplac... Jax is also going to be giving even lower-level access than TF, which may interest you. You did not give an example of something GPUs can't do. all you said was that TPUs are faster for a specific function in your case. Well yeah, I care about achieving goals in my specific case, as you do yours. And simply getting together a VM that can feed 500 examples/sec to a set of GPUs is a massive undertaking in and of itself. TPUs make it more or less "easy" in comparison. (I won't say effortless, since it does take some effort to get yourself into the TPU programming mindset.)
  - deleted 6y ago
    [deleted]
  - sorenbouma 6y ago
    I'm a TPU user and I'd be interested to see a specific example of something that can be done on TPU but not GPU. Perhaps I'm just not experienced enough with the programming model, but I've found them to be strictly less flexible/more tricky than GPUs, especially for things like conditional execution, multiple graphs, variable size inputs and custom ops.
    - sillysaurusx 6y ago
      Sure! I'd love to chat TPUs. There's a #tpu discord channel on the MLPerf discord: https://github.com/shawwn/tpunicorn#ml-community https://github.com/shawwn/tpunicorn#ml-community The central reason that TPUs feel less flexible is Google's awful mistake in encouraging everyone to use TPUEstimator as the One True API For Doing TPU Programming. Getting off that API was the single biggest boost to my TPU skills. You can see an example of how to do that here: https://github.com/shawwn/ml-notes/blob/master/train_runner.py https://github.com/shawwn/ml-notes/blob/master/train_runner.... This is a repo that can train GPT-2 1.5B at 10 examples/sec on a TPUv3-8 (aka around 10k tokens/sec). Happy to answer any specific questions or peek at codebases you're hoping to run on TPUs.
      - slaymaker1907 6y ago
        That doesn't answer the question of what a TPU can do that a GPU can't. I think the OP means impossible for the GPU, not just slower.
  - slaymaker1907 6y ago
    You can run basically any C program on a CUDA core even those requiring malloc. It may not be efficient but you can do it. Google themselves call GPUs general purpose and TPUs domain specific. https://cloud.google.com/blog/products/ai-machine-learning/what-makes-tpus-fine-tuned-for-deep-learning https://cloud.google.com/blog/products/ai-machine-learning/w...