5 ms·

This is pretty neat since it is the first time in years that a top-tier deep learning framework has official support for any training accelerator with open sour

This is pretty neat since it is the first time in years that a top-tier deep learning framework has official support for any training accelerator with open source kernel drivers. I guess the TPU also doesn't require kernel drivers because you talk to it over network instead of PCIE. But you cannot buy a TPU, only the int8 edge TPU is for sale. (And I've heard that the edge TPU's are absolutely top-notch for performance per $ and Watt right now, as an aside.)

deleted 6y ago
[deleted]
rfoo 6y ago
I believe TensorFlow is a top-tier deep learning framework, and it had ROCm support since 2018. > edge TPU's are absolutely top-notch for performance per $ and Watt right now Do you mean "aren't"? The performance per $ and Watt is not awesome even when it was released, I was hoping for great toolchain support but that also didn't happen.
- NavinF 6y ago
  Are there any <10W boards that have better performance/watt for object detection? If it exists, I wanna buy it
  - my123 6y ago
    Jetson Xavier NX, but that comes with a high price tag. It’s much more powerful however.
    - dheera 6y ago
      I'll also add a caveat that toolage for Jetson boards is extremely incomplete. They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. WTF, it's 2021. If you want to use anything remotely state-of-the-art like EfficientDet or HRNet or Deeplab or whatever you're left in the dark. Yes you can run TensorFlow or PyTorch (thankfully they give you wheels for those now; before you had to google "How to install TensorFlow on Jetson" and wade through hundreds of forum pages) but they're not as fast at inference.
      - my123 6y ago
        You have https://github.com/NVIDIA-AI-IOT/torch2trt https://github.com/NVIDIA-AI-IOT/torch2trt as an option for example to use your own models on TensorRT just fine. And https://github.com/tensorflow/tensorrt https://github.com/tensorflow/tensorrt for TF-TRT integration.
        codethief 6y ago
        TF-TRT doesn't work nearly as well as pure TRT. On my Jetson Nano a 300x300 SSD-MobileNetV2 with 2 object classes runs at 5 FPS using TF, <10 FPS using TF-TRT and 30 FPS using TensorRT.
        dheera 6y ago
        This. Try any recent network with TF-TRT and you'll find that memory is constantly being copied back and forth between TF and TRT components of the system every time it stumbles upon an operation not supported in TRT. As such I often got slower results with TF-TRT than just pure TF, and at most a marginal improvement, even though what TRT does is conceptually awesome from a deployment standpoint, and if it only supported all the operations in TF, it could be a several-fold speed up in many cases.
        codethief 6y ago
        > even though what TRT does is conceptually awesome from a deployment standpoint I thought the same until, earlier this week, I realized that if I convert a model to TensorRT and serialize it & store it in a file that file is specific to my device (i.e. my specific Jetson Nano), meaning that my colleagues can't run that file on their Jetson Nano. What the actual fuck. Do you happen to have found a workaround for this? I really don't want to have to convert the model anew every single time I deploy it. There are just too many moving parts involved in the conversion process, dependency-wise.
      - homarp 6y ago
        https://github.com/wang-xinyu/tensorrtx https://github.com/wang-xinyu/tensorrtx has a lot of models implemented for TensorRT. They test on GTX1080 not jetson nano though, so some work is also needed. TVM is another alternative to get models to inference fast on nano
  - geek_at 6y ago
    I used the Intel neural compute sticks [1] for my porn detection service [2] and they worked great. Could import and run models on a Pi with ease [1] https://blog.haschek.at/2018/fight-child-pornography-with-raspi-and-deep-learning.html https://blog.haschek.at/2018/fight-child-pornography-with-ra... [2] https://nsfw-categorize.it/ https://nsfw-categorize.it/
    - belval 6y ago
      Very interesting project and article. You should consider submitting it to HN for its own post (if you have not done so already).
- etaioinshrdlu 6y ago
  Tensorflow doesn't seem to officially support ROCm, only unofficial community projects do. This is official support from PyTorch.
  - rfoo 6y ago
    Tensorflow does officially support ROCm. The project was started by AMD and later upstreamed. https://github.com/tensorflow/tensorflow/tree/master/tensorflow/stream_executor/rocm https://github.com/tensorflow/tensorflow/tree/master/tensorf... https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/Dockerfile.rocm https://github.com/tensorflow/tensorflow/blob/master/tensorf... It is true that it is not Google who are distributing binaries compiled with ROCm support through PyPI (tensorflow and tensorflow-gpu is uploaded by Google, but tensorflow-rocm is uploaded by AMD). Is this what you meant by "not officially supporting"?
    - etaioinshrdlu 6y ago
      Oh, interesting. I do wonder if Google puts the same quality control and testing into the Rocm version, though. Otherwise it would really be a lower tier of official support. Granted, I don't know anything about the quality of PyTorch's support either.
    - t-vi 6y ago
      What you describe sounds a lot like the PyTorch support before this announcement: You could download PyTorch from AMD's ROCm site or build it yourself for >= 2 years now and this worked very reliably. (Edit: The two years (Nov 2018 or so) are the ones I can attest to from using it personally, but it probably didn't start then.) The news here is that the PyTorch team and AMD are confident enough about the quality that they're putting it on the front page. This has been a long way in the making, and finally achieving official support is a great step for the team working on it.
- dheera 6y ago
  Jetson Nano: 1.4 TOPS/W, Coral TPU: 2 TOPS/W ? Of course it doesn't really help that Google refuses to release a more powerful TPU that can compete with e.g. Xavier NX or a V100 or RTX3080 so for lots of applications there isn't much of a choice but to use NVIDIA.
  - rfoo 6y ago
    Sorry, should have mentioned "if you have access to Shenzhen" in my post :) What I have in mind is something like RK3399Pro, it has a proprietary NPU at roughly 3Tops / 1.5W (on paper). But its toolchain is rather hard to use. Hisilicon had similar offerings. There are also Kendryte K210 which claims 1Tops @ 0.3W but I haven't get any chance to try it. I was already playing with RK3399Pro When Edge TPU was announced, life is tough when you had to feed your model into a blackbox "model converter" from the vendor. That's the part I hope Edge TPU excels at. But months later I was greeted by... "to use Edge TPU, you have to upload your TFLite model to our online model optimizer", which is worse!
    - simcop2387 6y ago
      There's now a blackbox compiler that doesn't have to run on their service, but it's basically the same as all the others now because of that.
      - my123 6y ago
        On Xavier, the dedicated AI inference block is open source hardware. Available at http://nvdla.org/ http://nvdla.org/

deleted 6y ago

[deleted]

rfoo 6y ago

I believe TensorFlow is a top-tier deep learning framework, and it had ROCm support since 2018. > edge TPU's are absolutely top-notch for performance per $ and Watt right now Do you mean "aren't"? The performance per $ and Watt is not awesome even when it was released, I was hoping for great toolchain support but that also didn't happen.

NavinF 6y ago
Are there any <10W boards that have better performance/watt for object detection? If it exists, I wanna buy it
- my123 6y ago
  Jetson Xavier NX, but that comes with a high price tag. It’s much more powerful however.
  - dheera 6y ago
    I'll also add a caveat that toolage for Jetson boards is extremely incomplete. They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. WTF, it's 2021. If you want to use anything remotely state-of-the-art like EfficientDet or HRNet or Deeplab or whatever you're left in the dark. Yes you can run TensorFlow or PyTorch (thankfully they give you wheels for those now; before you had to google "How to install TensorFlow on Jetson" and wade through hundreds of forum pages) but they're not as fast at inference.
    - my123 6y ago
      You have https://github.com/NVIDIA-AI-IOT/torch2trt https://github.com/NVIDIA-AI-IOT/torch2trt as an option for example to use your own models on TensorRT just fine. And https://github.com/tensorflow/tensorrt https://github.com/tensorflow/tensorrt for TF-TRT integration.
      - codethief 6y ago
        TF-TRT doesn't work nearly as well as pure TRT. On my Jetson Nano a 300x300 SSD-MobileNetV2 with 2 object classes runs at 5 FPS using TF, <10 FPS using TF-TRT and 30 FPS using TensorRT.
        dheera 6y ago
        This. Try any recent network with TF-TRT and you'll find that memory is constantly being copied back and forth between TF and TRT components of the system every time it stumbles upon an operation not supported in TRT. As such I often got slower results with TF-TRT than just pure TF, and at most a marginal improvement, even though what TRT does is conceptually awesome from a deployment standpoint, and if it only supported all the operations in TF, it could be a several-fold speed up in many cases.
        codethief 6y ago
        > even though what TRT does is conceptually awesome from a deployment standpoint I thought the same until, earlier this week, I realized that if I convert a model to TensorRT and serialize it & store it in a file that file is specific to my device (i.e. my specific Jetson Nano), meaning that my colleagues can't run that file on their Jetson Nano. What the actual fuck. Do you happen to have found a workaround for this? I really don't want to have to convert the model anew every single time I deploy it. There are just too many moving parts involved in the conversion process, dependency-wise.
    - homarp 6y ago
      https://github.com/wang-xinyu/tensorrtx https://github.com/wang-xinyu/tensorrtx has a lot of models implemented for TensorRT. They test on GTX1080 not jetson nano though, so some work is also needed. TVM is another alternative to get models to inference fast on nano
- geek_at 6y ago
  I used the Intel neural compute sticks [1] for my porn detection service [2] and they worked great. Could import and run models on a Pi with ease [1] https://blog.haschek.at/2018/fight-child-pornography-with-raspi-and-deep-learning.html https://blog.haschek.at/2018/fight-child-pornography-with-ra... [2] https://nsfw-categorize.it/ https://nsfw-categorize.it/
  - belval 6y ago
    Very interesting project and article. You should consider submitting it to HN for its own post (if you have not done so already).
etaioinshrdlu 6y ago
Tensorflow doesn't seem to officially support ROCm, only unofficial community projects do. This is official support from PyTorch.
- rfoo 6y ago
  Tensorflow does officially support ROCm. The project was started by AMD and later upstreamed. https://github.com/tensorflow/tensorflow/tree/master/tensorflow/stream_executor/rocm https://github.com/tensorflow/tensorflow/tree/master/tensorf... https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/Dockerfile.rocm https://github.com/tensorflow/tensorflow/blob/master/tensorf... It is true that it is not Google who are distributing binaries compiled with ROCm support through PyPI (tensorflow and tensorflow-gpu is uploaded by Google, but tensorflow-rocm is uploaded by AMD). Is this what you meant by "not officially supporting"?
  - etaioinshrdlu 6y ago
    Oh, interesting. I do wonder if Google puts the same quality control and testing into the Rocm version, though. Otherwise it would really be a lower tier of official support. Granted, I don't know anything about the quality of PyTorch's support either.
  - t-vi 6y ago
    What you describe sounds a lot like the PyTorch support before this announcement: You could download PyTorch from AMD's ROCm site or build it yourself for >= 2 years now and this worked very reliably. (Edit: The two years (Nov 2018 or so) are the ones I can attest to from using it personally, but it probably didn't start then.) The news here is that the PyTorch team and AMD are confident enough about the quality that they're putting it on the front page. This has been a long way in the making, and finally achieving official support is a great step for the team working on it.
dheera 6y ago
Jetson Nano: 1.4 TOPS/W, Coral TPU: 2 TOPS/W ? Of course it doesn't really help that Google refuses to release a more powerful TPU that can compete with e.g. Xavier NX or a V100 or RTX3080 so for lots of applications there isn't much of a choice but to use NVIDIA.
- rfoo 6y ago
  Sorry, should have mentioned "if you have access to Shenzhen" in my post :) What I have in mind is something like RK3399Pro, it has a proprietary NPU at roughly 3Tops / 1.5W (on paper). But its toolchain is rather hard to use. Hisilicon had similar offerings. There are also Kendryte K210 which claims 1Tops @ 0.3W but I haven't get any chance to try it. I was already playing with RK3399Pro When Edge TPU was announced, life is tough when you had to feed your model into a blackbox "model converter" from the vendor. That's the part I hope Edge TPU excels at. But months later I was greeted by... "to use Edge TPU, you have to upload your TFLite model to our online model optimizer", which is worse!
  - simcop2387 6y ago
    There's now a blackbox compiler that doesn't have to run on their service, but it's basically the same as all the others now because of that.
    - my123 6y ago
      On Xavier, the dedicated AI inference block is open source hardware. Available at http://nvdla.org/ http://nvdla.org/