4 ms·

what are most people asking for - pytorch? or are they leaving those decisions up to you?

m0zg 6y ago
As a practitioner in the field and in similar circumstances, the answer will probably be: it doesn't really matter. Personally I prefer PyTorch, but I can work with whatever the client already uses, if anything. For one of my clients I worked in both because there was no viable deployment option out of PyTorch to one of their platforms. There still isn't, so both models are still maintained and trained. If they aren't 100% set in their ways, I do make them aware that things will move at about half the speed with TF, so they'll effectively be paying twice as much. If they are set in their ways, I do not mention it, since I'm not going to change their mind anyway. That said PyTorch 1.5.0 is just broken pretty much - tensor permute (which in computer vision you end up doing for every input tensor) is 10x slower than it used to be. There's an issue in GitHub already. I'm beginning to worry about PyTorch.
- m3at 6y ago
  Thanks for sharing your experience! I'm working with TF and Pytorch as well, but so far for the later I have found the project to be reasonably reliable (though I did find Chainer considerably more polished). Can you share more about what worries you with Pytorch?
  - m0zg 6y ago
    1.5.0 is basically broken for computer vision. Input tensors are usually in NHWC memory format, but PyTorch (and CUDA) prefers NCHW (planar). So you'd normally run permute() to move things around. But https://github.com/pytorch/pytorch/issues/37142 https://github.com/pytorch/pytorch/issues/37142 (and possibly other bugs) kinda gets in the way of that. I had to roll back, since training got way slower than it was before, and it wasn't super fast to begin with, even on my quad-GPU workstation. The fact that such obvious, severe bugs make it through the release process likely means that there isn't really much of a release process. And what's in place doesn't even test the release on totally bread-and-butter models like resnet50.
    - smhx 6y ago
      PyTorch maintainer here: we're looking into that and if needed will issue 1.5.1 asap. It didn't show up in release testing, which among other things does end-to-end imagenet runs with ResNet50 and a few other models (i.e. time and memory didn't regress). Will also figure out how to catch this early.
      - smhx 6y ago
        @m0zg if you could comment on the issue you quoted with any details, it would be really helpful to us. Unlike what is reported in the github issue, `permute` isn't the regression. For reference, one of the core devs added more details based on where we are with our investigation: https://github.com/pytorch/pytorch/issues/37142#issuecomment-623205729 https://github.com/pytorch/pytorch/issues/37142#issuecomment...

m0zg 6y ago

As a practitioner in the field and in similar circumstances, the answer will probably be: it doesn't really matter. Personally I prefer PyTorch, but I can work with whatever the client already uses, if anything. For one of my clients I worked in both because there was no viable deployment option out of PyTorch to one of their platforms. There still isn't, so both models are still maintained and trained. If they aren't 100% set in their ways, I do make them aware that things will move at about half the speed with TF, so they'll effectively be paying twice as much. If they are set in their ways, I do not mention it, since I'm not going to change their mind anyway. That said PyTorch 1.5.0 is just broken pretty much - tensor permute (which in computer vision you end up doing for every input tensor) is 10x slower than it used to be. There's an issue in GitHub already. I'm beginning to worry about PyTorch.

m3at 6y ago
Thanks for sharing your experience! I'm working with TF and Pytorch as well, but so far for the later I have found the project to be reasonably reliable (though I did find Chainer considerably more polished). Can you share more about what worries you with Pytorch?
- m0zg 6y ago
  1.5.0 is basically broken for computer vision. Input tensors are usually in NHWC memory format, but PyTorch (and CUDA) prefers NCHW (planar). So you'd normally run permute() to move things around. But https://github.com/pytorch/pytorch/issues/37142 https://github.com/pytorch/pytorch/issues/37142 (and possibly other bugs) kinda gets in the way of that. I had to roll back, since training got way slower than it was before, and it wasn't super fast to begin with, even on my quad-GPU workstation. The fact that such obvious, severe bugs make it through the release process likely means that there isn't really much of a release process. And what's in place doesn't even test the release on totally bread-and-butter models like resnet50.
  - smhx 6y ago
    PyTorch maintainer here: we're looking into that and if needed will issue 1.5.1 asap. It didn't show up in release testing, which among other things does end-to-end imagenet runs with ResNet50 and a few other models (i.e. time and memory didn't regress). Will also figure out how to catch this early.
    - smhx 6y ago
      @m0zg if you could comment on the issue you quoted with any details, it would be really helpful to us. Unlike what is reported in the github issue, `permute` isn't the regression. For reference, one of the core devs added more details based on where we are with our investigation: https://github.com/pytorch/pytorch/issues/37142#issuecomment-623205729 https://github.com/pytorch/pytorch/issues/37142#issuecomment...