4 ms·

Jetson Xavier NX, but that comes with a high price tag. It’s much more powerful however.

by my123 6y ago

dheera 6y ago
I'll also add a caveat that toolage for Jetson boards is extremely incomplete. They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. WTF, it's 2021. If you want to use anything remotely state-of-the-art like EfficientDet or HRNet or Deeplab or whatever you're left in the dark. Yes you can run TensorFlow or PyTorch (thankfully they give you wheels for those now; before you had to google "How to install TensorFlow on Jetson" and wade through hundreds of forum pages) but they're not as fast at inference.
- my123 6y ago
  You have https://github.com/NVIDIA-AI-IOT/torch2trt https://github.com/NVIDIA-AI-IOT/torch2trt as an option for example to use your own models on TensorRT just fine. And https://github.com/tensorflow/tensorrt https://github.com/tensorflow/tensorrt for TF-TRT integration.
  - codethief 6y ago
    TF-TRT doesn't work nearly as well as pure TRT. On my Jetson Nano a 300x300 SSD-MobileNetV2 with 2 object classes runs at 5 FPS using TF, <10 FPS using TF-TRT and 30 FPS using TensorRT.
    - dheera 6y ago
      This. Try any recent network with TF-TRT and you'll find that memory is constantly being copied back and forth between TF and TRT components of the system every time it stumbles upon an operation not supported in TRT. As such I often got slower results with TF-TRT than just pure TF, and at most a marginal improvement, even though what TRT does is conceptually awesome from a deployment standpoint, and if it only supported all the operations in TF, it could be a several-fold speed up in many cases.
      - codethief 6y ago
        > even though what TRT does is conceptually awesome from a deployment standpoint I thought the same until, earlier this week, I realized that if I convert a model to TensorRT and serialize it & store it in a file that file is specific to my device (i.e. my specific Jetson Nano), meaning that my colleagues can't run that file on their Jetson Nano. What the actual fuck. Do you happen to have found a workaround for this? I really don't want to have to convert the model anew every single time I deploy it. There are just too many moving parts involved in the conversion process, dependency-wise.
- homarp 6y ago
  https://github.com/wang-xinyu/tensorrtx https://github.com/wang-xinyu/tensorrtx has a lot of models implemented for TensorRT. They test on GTX1080 not jetson nano though, so some work is also needed. TVM is another alternative to get models to inference fast on nano
  - codethief 6y ago
    How does TVM compare to TensorRT performance-wise?
    - homarp 6y ago
      it depends of your model https://forums.developer.nvidia.com/t/tensorrt-vs-tvm/79964 https://forums.developer.nvidia.com/t/tensorrt-vs-tvm/79964 https://discuss.tvm.apache.org/t/which-is-better-tvm-or-tensorrt/6736 https://discuss.tvm.apache.org/t/which-is-better-tvm-or-tens...
- codethief 6y ago
  > I'll also add a caveat that toolage for Jetson boards is extremely incomplete. A hundred times this. I was about to write another rant here but I already did that[0] a while ago, so I'll save my breath this time. :) Another fun fact regarding toolage: Today I discovered that many USB cameras work poorly on Jetsons (at least when using OpenCV), probably due to different drivers and/or the fact that OpenCV doesn't support ARM64 as well as it does x86_64. :( > They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. They supply you with such models? That's news to me. AFAIK converting something like SSD-MobileNetv2 from TensorFlow to TensorRT still requires substantial manual work and magic, as this code[1] attests to. There are countless (countless!) posts on the Nvidia forums by people complaining that they're not able to convert their models. [0]: https://news.ycombinator.com/item?id=26004235 https://news.ycombinator.com/item?id=26004235 [1]: https://github.com/jkjung-avt/tensorrt_demos/blob/master/ssd/build_engine.py https://github.com/jkjung-avt/tensorrt_demos/blob/master/ssd... (In fact, this is the only piece of code I've found on the entire internet that managed to successfully convert my SSD-MobileNetV2.)
  - dheera 6y ago
    They provide some SSD-Mobilenet-v2 here: https://github.com/dusty-nv/jetson-inference https://github.com/dusty-nv/jetson-inference Yeah, it works. I get 140 fps on a Xavier NX. It's super impressive for the wattage and size of the device. But they want you to train it using their horrid "DIGITS" interface, and it doesn't support any more recent networks. I really wish Nvidia would stop trying to reinvent the wheel in training and focus on keeping up with being able to properly parse all the operations in the latest state-of-the-art networks which are almost always in Pytorch or TF 2.x.
    - codethief 6y ago
      > They provide some SSD-Mobilenet-v2 here: https://github.com/dusty-nv/jetson-inference https://github.com/dusty-nv/jetson-inference I was aware of that repository but from taking a cursory look at it I had thought dusty was just converting models from PyTorch to TensorRT, like here[0, 1]. Am I missing something? (EDIT: Oh, never mind. You probably meant the model trained on COCO[2]. Now I remember that I ignored it way back when because I needed much better accuracy.) > I get 140 fps on a Xavier NX That really is impressive. Holy shit. [0]: https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-ssd.md https://github.com/dusty-nv/jetson-inference/blob/master/doc... [1]: https://github.com/dusty-nv/jetson-inference/issues/896#issuecomment-767033699 https://github.com/dusty-nv/jetson-inference/issues/896#issu... [2]: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md#downloading-models https://github.com/dusty-nv/jetson-inference/blob/master/doc...

Jetson Xavier NX, but that comes with a high price tag. It’s much more powerful however.

by my123 6y ago

Jetson Xavier NX, but that comes with a high price tag. It’s much more powerful however.

I'll also add a caveat that toolage for Jetson boards is extremely incomplete. They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. WTF, it's 2021. If you want to use anything remotely state-of-the-art like EfficientDet or HRNet or Deeplab or whatever you're left in the dark. Yes you can run TensorFlow or PyTorch (thankfully they give you wheels for those now; before you had to google "How to install TensorFlow on Jetson" and wade through hundreds of forum pages) but they're not as fast at inference.

my123 6y ago
You have https://github.com/NVIDIA-AI-IOT/torch2trt https://github.com/NVIDIA-AI-IOT/torch2trt as an option for example to use your own models on TensorRT just fine. And https://github.com/tensorflow/tensorrt https://github.com/tensorflow/tensorrt for TF-TRT integration.
- codethief 6y ago
  TF-TRT doesn't work nearly as well as pure TRT. On my Jetson Nano a 300x300 SSD-MobileNetV2 with 2 object classes runs at 5 FPS using TF, <10 FPS using TF-TRT and 30 FPS using TensorRT.
  - dheera 6y ago
    This. Try any recent network with TF-TRT and you'll find that memory is constantly being copied back and forth between TF and TRT components of the system every time it stumbles upon an operation not supported in TRT. As such I often got slower results with TF-TRT than just pure TF, and at most a marginal improvement, even though what TRT does is conceptually awesome from a deployment standpoint, and if it only supported all the operations in TF, it could be a several-fold speed up in many cases.
    - codethief 6y ago
      > even though what TRT does is conceptually awesome from a deployment standpoint I thought the same until, earlier this week, I realized that if I convert a model to TensorRT and serialize it & store it in a file that file is specific to my device (i.e. my specific Jetson Nano), meaning that my colleagues can't run that file on their Jetson Nano. What the actual fuck. Do you happen to have found a workaround for this? I really don't want to have to convert the model anew every single time I deploy it. There are just too many moving parts involved in the conversion process, dependency-wise.
homarp 6y ago
https://github.com/wang-xinyu/tensorrtx https://github.com/wang-xinyu/tensorrtx has a lot of models implemented for TensorRT. They test on GTX1080 not jetson nano though, so some work is also needed. TVM is another alternative to get models to inference fast on nano
- codethief 6y ago
  How does TVM compare to TensorRT performance-wise?
  - homarp 6y ago
    it depends of your model https://forums.developer.nvidia.com/t/tensorrt-vs-tvm/79964 https://forums.developer.nvidia.com/t/tensorrt-vs-tvm/79964 https://discuss.tvm.apache.org/t/which-is-better-tvm-or-tensorrt/6736 https://discuss.tvm.apache.org/t/which-is-better-tvm-or-tens...
codethief 6y ago
> I'll also add a caveat that toolage for Jetson boards is extremely incomplete. A hundred times this. I was about to write another rant here but I already did that[0] a while ago, so I'll save my breath this time. :) Another fun fact regarding toolage: Today I discovered that many USB cameras work poorly on Jetsons (at least when using OpenCV), probably due to different drivers and/or the fact that OpenCV doesn't support ARM64 as well as it does x86_64. :( > They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. They supply you with such models? That's news to me. AFAIK converting something like SSD-MobileNetv2 from TensorFlow to TensorRT still requires substantial manual work and magic, as this code[1] attests to. There are countless (countless!) posts on the Nvidia forums by people complaining that they're not able to convert their models. [0]: https://news.ycombinator.com/item?id=26004235 https://news.ycombinator.com/item?id=26004235 [1]: https://github.com/jkjung-avt/tensorrt_demos/blob/master/ssd/build_engine.py https://github.com/jkjung-avt/tensorrt_demos/blob/master/ssd... (In fact, this is the only piece of code I've found on the entire internet that managed to successfully convert my SSD-MobileNetV2.)
- dheera 6y ago
  They provide some SSD-Mobilenet-v2 here: https://github.com/dusty-nv/jetson-inference https://github.com/dusty-nv/jetson-inference Yeah, it works. I get 140 fps on a Xavier NX. It's super impressive for the wattage and size of the device. But they want you to train it using their horrid "DIGITS" interface, and it doesn't support any more recent networks. I really wish Nvidia would stop trying to reinvent the wheel in training and focus on keeping up with being able to properly parse all the operations in the latest state-of-the-art networks which are almost always in Pytorch or TF 2.x.
  - codethief 6y ago
    > They provide some SSD-Mobilenet-v2 here: https://github.com/dusty-nv/jetson-inference https://github.com/dusty-nv/jetson-inference I was aware of that repository but from taking a cursory look at it I had thought dusty was just converting models from PyTorch to TensorRT, like here[0, 1]. Am I missing something? (EDIT: Oh, never mind. You probably meant the model trained on COCO[2]. Now I remember that I ignored it way back when because I needed much better accuracy.) > I get 140 fps on a Xavier NX That really is impressive. Holy shit. [0]: https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-ssd.md https://github.com/dusty-nv/jetson-inference/blob/master/doc... [1]: https://github.com/dusty-nv/jetson-inference/issues/896#issuecomment-767033699 https://github.com/dusty-nv/jetson-inference/issues/896#issu... [2]: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md#downloading-models https://github.com/dusty-nv/jetson-inference/blob/master/doc...