5 ms·

I dislike that pytorch advertises TPU support. Pytorch doesn’t support TPUs. Pytorch supports a gimped version of TPUs that have no access to the TPU CPU, a mas

I dislike that pytorch advertises TPU support. Pytorch doesn’t support TPUs. Pytorch supports a gimped version of TPUs that have no access to the TPU CPU, a massive 300GB memory store that handles infeed. No infeed means you have to feed the TPUs manually, on demand, like a gpu. And TPUs are not GPUs. When you try to do that, you’re talking at least a 40x slowdown, no exaggeration. The TPU CPU is the heart of the TPU’s power and advantage over GPUs, and neither pytorch nor Jax support it at all yet. No MLPerf benchmark will ever use pytorch in its current form on TPUs. Luckily, that form is changing. There are interesting plans. But they are still just plans. It’s better to go the other direction, I think. I ported pytorch to tensorflow: https://twitter.com/theshawwn/status/1311925180126511104?s=21 https://twitter.com/theshawwn/status/1311925180126511104?s=2... Pytorch is mostly just an api. And that api is mostly python. When people say they “like pytorch”, they’re expressing a preference for how to organize ML code, not for the set of operations available to you when you use pytorch.

wfalcon 6y ago
We currently have google engineers training on TPU pods with PyTorch Lightning. TPU support is VERY real... but yes, sometimes it breaks but PyTorch and Google are working very hard to bridge that gap. But we have dedicated partners at Google on the TPU team working to get Lightning working seamlessly on pods. Check out the discussions here: https://github.com/PyTorchLightning/pytorch-lightning/issues/3058 https://github.com/PyTorchLightning/pytorch-lightning/issues...
- sillysaurusx 6y ago
  No, you do not support the TPU infeed, and this is a crucial distinction. Saying that you do support this has caused endless confusion and much surprise. It’s almost not an exaggeration to say that you’re lying (sorry for phrasing this so bluntly, but I’ve seriously spent dozens of hours trying to break this misconception due to hype like this). TPU support is real. Pytorch does in fact run on TPUs. But you don’t support TPU CPU memory, the staging area that you’re supposed to fill with training data. That staging area is why a TPU v3-512 pod can train an imagenet resnet classifier in 3 minutes at around 1M examples per second. You will not get anywhere near that performance with pytorch on TPUs. In fact, you’re expected to create a separate VM for every 8 TPU cores. The VMs are in charge of feeding the cores. That’s insane; I’ve driven TPU pods from a single n1-standard-2 using tensorflow. Repeat after me: if you are required to create more than one VM, you do not (yet!) support TPU pods. I wish I could triple underline this and put it in bold. People need to understand the limitations of this technique. Creating 256 VMs to feed a v3-2048 is not sustainable.
  - minimaxir 6y ago
    There's a difference between "supporting TPUs" and "supporting TPUs at 100% potential". Although the distinction is important, I don't think the marketing here is misleading.
    - sillysaurusx 6y ago
      Not only is it misleading, it even somehow tricked you. :) We’re not talking about a small 10% reduction in performance here. We’re talking like 40x differences. If it seems unbelievable, and like it can’t possibly be true, well: now you understand my frustration here, and why I’m trying to break the myth. Notice not a single benchmark has ever gone head to head in MLPerf using pytorch on TPUs. And that’s because using pytorch on TPUs requires you to feed each image manually to the TPU on demand, from your VM. Meaning the TPU is always infeed bound. Engineers should be wincing at the sound of that. Especially anyone with graphics experience. Being infeed bound means you have lots of horsepower sitting around doing nothing. And that’s exactly the situation you’ll end up in with this technique. There’s a way to settle this decisively: train a resnet classifier on imagenet, as quickly as possible. If you get anywhere near the MLPerf v0.6 benchmarks for tensorflow on TPUs, I will instantly pivot the other direction and sing the praises of pytorch on TPUs far and wide.
      - deleted 6y ago
        [deleted]
  - wfalcon 6y ago
    Like I said... pytorch and tensorflow team are working very hard to make this work. And yes, it's not a 1:1 with tensorflow, but we're making progress very aggressively.
    - sillysaurusx 6y ago
      I love what you guys are doing, and I love improving the ML ecosystem, but you’ve godda understand, people see this and think “oh, ok, it’s a small difference, no big deal.” In fact it’s a huge difference. Picture a person with one arm and without legs. Would you say they aren’t “1:1 in terms of features”? They certainly won’t be winning any races. And unlike real people, you can’t graft on a prosthetic limb to help this situation. The issue I’m describing here is a fundamental one that everyone keeps trying to sweep under the rug and pretend isn’t an issue. And then everyone wonders what’s going on.
arugulum 6y ago
I want to signal-boost this. TPU support on PyTorch is partial. You can run modeling computation on the TPU with PyTorch, but not the data-loading. And without the TPU's data-loading, you're significantly, significantly bottle-necked, to the point where you are often better off using GPUs. The reason why TensorFlow and TPUs synergize so well is that the TPUs themselves can consume data for training, allowing for massive scalability. I have great respect for the PyTorch-TPU team, but I would recommend not heavily advertising PyTorch-TPU support until this major feature disparity is made up.
- wfalcon 6y ago
  we highlight these issues in our docs explicitly. https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#tpu-support https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#...
  - marcinzm 6y ago
    Can you point out where exactly in those docs you highlight the issue? I just read the linked page and found no references to data loading limitations or performance limitations. Is it only in the video which isn't search indexed and few people would bother watching? edit: The page literally advertises the speed of TPUs with "In general, a single TPU is about as fast as 5 V100 GPUs!" which is the exact opposite of warning people.
hobofan 6y ago
> When people say they “like pytorch”, they’re expressing a preference for how to organize ML code Maybe that's the way you feel, but for me that's very different. Pytorch is much more than just an API (which is also nothing to scoff at). It's also much cleaner documentation, a very different ecosystem of libraries (mostly better, but sometimes lacking depending on the niche), less magic (which makes it easier to debug). It also has the benefit of less ecosystem churn, while the transitions of TF1->2 as well as the external Keras->internal Keras are a shitshow that's almost as bad as Python2->3.
- desku 6y ago
  What niche libraries do you think PyTorch is lacking? Do you have some examples of ones that exist in Tensorflow with no PyTorch equivalent?
  - hobofan 6y ago
    I meant model libraries/implementations with that like e.g. Stardist[0]. Obviously there will be some research niches that are mostly implemented in TF over PyTorch or the other way around. [0]: https://github.com/mpicbg-csbd/stardist https://github.com/mpicbg-csbd/stardist
tmabraham 6y ago
We both know (and correct me if I'm wrong) that this is an issue with the GCP cloud architecture. https://github.com/pytorch/xla/issues/1858 https://github.com/pytorch/xla/issues/1858 So you can't blame the PyTorch team. If there's anyone to blame, it's Google Cloud. In the meantime, I don't think there's any harm advertising PyTorch with TPU support if running on TPUs with PyTorch is often much faster than running on GPUs with PyTorch.
- tmabraham 6y ago
  In the above-linked GitHub issue, the Google TPU team is now giving an ETA of early 2021. At that point PyTorch TPU training (including on TPU pods) should be equivalent to TF TPU training. But I think my point still stands that as long as PyTorch TPU training is faster than GPU training, even in its current state, there's nothing wrong advertising TPU support now.

wfalcon 6y ago

We currently have google engineers training on TPU pods with PyTorch Lightning. TPU support is VERY real... but yes, sometimes it breaks but PyTorch and Google are working very hard to bridge that gap. But we have dedicated partners at Google on the TPU team working to get Lightning working seamlessly on pods. Check out the discussions here: https://github.com/PyTorchLightning/pytorch-lightning/issues/3058 https://github.com/PyTorchLightning/pytorch-lightning/issues...

sillysaurusx 6y ago
No, you do not support the TPU infeed, and this is a crucial distinction. Saying that you do support this has caused endless confusion and much surprise. It’s almost not an exaggeration to say that you’re lying (sorry for phrasing this so bluntly, but I’ve seriously spent dozens of hours trying to break this misconception due to hype like this). TPU support is real. Pytorch does in fact run on TPUs. But you don’t support TPU CPU memory, the staging area that you’re supposed to fill with training data. That staging area is why a TPU v3-512 pod can train an imagenet resnet classifier in 3 minutes at around 1M examples per second. You will not get anywhere near that performance with pytorch on TPUs. In fact, you’re expected to create a separate VM for every 8 TPU cores. The VMs are in charge of feeding the cores. That’s insane; I’ve driven TPU pods from a single n1-standard-2 using tensorflow. Repeat after me: if you are required to create more than one VM, you do not (yet!) support TPU pods. I wish I could triple underline this and put it in bold. People need to understand the limitations of this technique. Creating 256 VMs to feed a v3-2048 is not sustainable.
- minimaxir 6y ago
  There's a difference between "supporting TPUs" and "supporting TPUs at 100% potential". Although the distinction is important, I don't think the marketing here is misleading.
  - sillysaurusx 6y ago
    Not only is it misleading, it even somehow tricked you. :) We’re not talking about a small 10% reduction in performance here. We’re talking like 40x differences. If it seems unbelievable, and like it can’t possibly be true, well: now you understand my frustration here, and why I’m trying to break the myth. Notice not a single benchmark has ever gone head to head in MLPerf using pytorch on TPUs. And that’s because using pytorch on TPUs requires you to feed each image manually to the TPU on demand, from your VM. Meaning the TPU is always infeed bound. Engineers should be wincing at the sound of that. Especially anyone with graphics experience. Being infeed bound means you have lots of horsepower sitting around doing nothing. And that’s exactly the situation you’ll end up in with this technique. There’s a way to settle this decisively: train a resnet classifier on imagenet, as quickly as possible. If you get anywhere near the MLPerf v0.6 benchmarks for tensorflow on TPUs, I will instantly pivot the other direction and sing the praises of pytorch on TPUs far and wide.
    - deleted 6y ago
      [deleted]
- wfalcon 6y ago
  Like I said... pytorch and tensorflow team are working very hard to make this work. And yes, it's not a 1:1 with tensorflow, but we're making progress very aggressively.
  - sillysaurusx 6y ago
    I love what you guys are doing, and I love improving the ML ecosystem, but you’ve godda understand, people see this and think “oh, ok, it’s a small difference, no big deal.” In fact it’s a huge difference. Picture a person with one arm and without legs. Would you say they aren’t “1:1 in terms of features”? They certainly won’t be winning any races. And unlike real people, you can’t graft on a prosthetic limb to help this situation. The issue I’m describing here is a fundamental one that everyone keeps trying to sweep under the rug and pretend isn’t an issue. And then everyone wonders what’s going on.

arugulum 6y ago

I want to signal-boost this. TPU support on PyTorch is partial. You can run modeling computation on the TPU with PyTorch, but not the data-loading. And without the TPU's data-loading, you're significantly, significantly bottle-necked, to the point where you are often better off using GPUs. The reason why TensorFlow and TPUs synergize so well is that the TPUs themselves can consume data for training, allowing for massive scalability. I have great respect for the PyTorch-TPU team, but I would recommend not heavily advertising PyTorch-TPU support until this major feature disparity is made up.

wfalcon 6y ago
we highlight these issues in our docs explicitly. https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#tpu-support https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#...
- marcinzm 6y ago
  Can you point out where exactly in those docs you highlight the issue? I just read the linked page and found no references to data loading limitations or performance limitations. Is it only in the video which isn't search indexed and few people would bother watching? edit: The page literally advertises the speed of TPUs with "In general, a single TPU is about as fast as 5 V100 GPUs!" which is the exact opposite of warning people.

hobofan 6y ago

> When people say they “like pytorch”, they’re expressing a preference for how to organize ML code Maybe that's the way you feel, but for me that's very different. Pytorch is much more than just an API (which is also nothing to scoff at). It's also much cleaner documentation, a very different ecosystem of libraries (mostly better, but sometimes lacking depending on the niche), less magic (which makes it easier to debug). It also has the benefit of less ecosystem churn, while the transitions of TF1->2 as well as the external Keras->internal Keras are a shitshow that's almost as bad as Python2->3.

desku 6y ago
What niche libraries do you think PyTorch is lacking? Do you have some examples of ones that exist in Tensorflow with no PyTorch equivalent?
- hobofan 6y ago
  I meant model libraries/implementations with that like e.g. Stardist[0]. Obviously there will be some research niches that are mostly implemented in TF over PyTorch or the other way around. [0]: https://github.com/mpicbg-csbd/stardist https://github.com/mpicbg-csbd/stardist

tmabraham 6y ago

We both know (and correct me if I'm wrong) that this is an issue with the GCP cloud architecture. https://github.com/pytorch/xla/issues/1858 https://github.com/pytorch/xla/issues/1858 So you can't blame the PyTorch team. If there's anyone to blame, it's Google Cloud. In the meantime, I don't think there's any harm advertising PyTorch with TPU support if running on TPUs with PyTorch is often much faster than running on GPUs with PyTorch.

tmabraham 6y ago
In the above-linked GitHub issue, the Google TPU team is now giving an ETA of early 2021. At that point PyTorch TPU training (including on TPU pods) should be equivalent to TF TPU training. But I think my point still stands that as long as PyTorch TPU training is faster than GPU training, even in its current state, there's nothing wrong advertising TPU support now.