5 ms·

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm st

I've done this with a Jetson Xavier, 4 CCTV cameras and a PoE hub. You really want to use DeepStream and C/C++ for inference, not Python and TensorFlow. I'm streaming ~20 fps (17 to 30) 720P directly from my home IP4 address, and when a person is in-frame long enough and caught by the tracker, a stream goes to an AWS endpoint for storage. I've experimented with both SSDMobileNet and Yolo3, which are both pretty error prone but they do a much better job filtering out moving tree limbs and passing clouds, unlike Arlo. You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I wrote very little of the code, honestly: only the capture pipe required a new C element. I started with NVidia DeepStream which is phenomenally well-written, and their built-in accelerated RTSP element, and added a custom GStreamer element that outputs a downsampled MPEG capture to the cloud when the upstream detector tracks an object. NVidia also wrote the tracker, you just need to provide an object detector like SSDMobileNet or YOLO. NVidia gets it. The main 4 camera-pipe mux splits into the AI engine and into a tee to the RTSP server on one side and my capture element on the other side. It was amazingly simple, and If I turn the CCD cameras down to 720P with h265 and a low bitrate, I don't need to turn on the noisy Xavier fan. The onboard Arm core does the detected downsampling (one camera only, a limitation right now) and pushes the video with a rest endpoint on a node server in the AWS cloud. I'm very pleased with it, I haven't tested scaling but if I turned off the GPU governors I could easily go to 8 cameras. I went with PoE because WiFi can't handle the demand.

exhilaration 6y ago
Do you have any links you could share to build something like this?
reichardt 6y ago
TensorFlow Lite with SSDLite-MobileNet gets you around 4 fps on a Raspberry Pi 4 (23 fps with a Coral USB Accelerator): https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi/blob/master/Raspberry_Pi_Guide.md https://github.com/EdjeElectronics/TensorFlow-Lite-Object-De...
- scottlamb 6y ago
  You should be able to do a lot better than that if you're careful with the software. As I mentioned in another comment, the Coral USB Accelerator can do at least 100 fps. I haven't looked closely at that link, but likely they're doing H.264 decoding in software using one thread, then downsampling in software using one thread, then waiting for the Coral USB accelerator, and repeating. Maybe they also have the accelerator plugged into a USB 2.0 port rather than a USB 3.0 port. The better approach is to use threading to keep all the Pi's cores busy and the USB accelerator busy at the same time, and to use hardware acceleration.
thebruce87m 6y ago
I’d love to read a write-up of this.
- staycoolboy 6y ago
  Me too. :) It is literally 90% in the NVIDIA SDK already. The demo examples provided with the kit read an HD stream, run inference and tracking -AND- provide an RTSP output! I started by replacing the HD stream with a videomux from my cameras into a composite image. Working with the the Gstreamer RTSP server is hard, and NVIDIA basically hands it to you. The next steps were to put tee elements on all 4 input streams to a selection demux and recompiled the tracker to send a bus event downstream to my element that controls the output of a video mux. This decides whether to send images to the final "videoconvert ! mp4mux ! payloder ! udpsink" pipeline that goes to a file... sort of. It gets a little messy because I couldn't figure out how to start / stop gst's filesink so I send it to a UDP port instead and have another process on the machine grab payload packets and decide when to create a new file. It used to be one big MP4 file that was pushed to the cloud, and .. um, the program would crash and I would restart the process to get the next file ... i know ... currently I'm trying to chop it up based on idle time (e.g., no new frames in 300ms? start a new file!). It's ugly and I still get corrupt files sometimes, which is why I haven't written it up... and i'm lazy. I bet I could fix this if there was a manual to RTFM, but GStreamer is such a bear to work with and the only source of help is their weird mailing list archive.
scottlamb 6y ago
> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" because I haven't tested on this exact setup yet. My Macbook Pro and Intel NUC are a lot more pleasant to experiment on (much faster compilation times). A few notes: * I'm currently just using the coral.ai prebuilt 300x300 MobileNet SSD v2 models. I haven't done much testing but can see it has notable false negatives and positives. It'd be wonderful to put together some shared training data [1] to use for transfer learning. I think then results could be much better. Anyone interested in starting something? I'd be happy to contribute! * iirc, I got the Coral USB Accelerator to do about 180 fps with this model. [edit: but don't trust my memory—it could have been as low as 100 fps.] It's easy enough to run the detection at a lower frame rate than the input as well—do the H.264 decoding on every frame but only do inference at fixed pts intervals. * You can also attach multiple Coral USB Accelerators to one system and make use of all of them. * Decoding the 8 streams is likely possible on the Pi 4 depending on your resolution. I haven't messed with this yet, but I think it might even be possible in software, and the Pi has hardware H.264 decoding that I haven't tried to use yet. * I use my cameras' 704x480 "sub" streams for motion detection and downsample that full image to the model's expected 300x300 input. Apparently some people do things like multiple inference against tiles of the image or running a second round of inference against a zoomed-in object detection region to improve confidence. That obviously increases the demand on both the CPU and TPU. * The Orange Pi AI Stick Lite is crazy cheap ($20) and supposedly comparable to the Coral USB Accelerator in speed. At that price if it works buying one per camera doesn't sound too crazy. But I'm not sure if drivers/toolchain support are any good. I have a PLAI Plug (basically the same thing but sold by the manufacturer). The PyTorch-based image classification on a prebuilt model works fine. I don't have the software to build models or do object detection so it's basically useless right now. They want to charge an unknown price for the missing software, but I think Orange Pi's rebrand might include it with the device? [1] https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc/m/rreU83FBBAAJ https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc...
- 6y ago
haser_au 6y ago
Similar to other's comments, I'd love to read a write up about this.
bufferoverflow 6y ago
I think I was able to run Yolo 3 on my shitty $99 smartphone a while ago. Did it for human detection. Don't remember the FPS, but it wasn't 0.1, it was much better than that. The beauty of a smartphone is it's all in one small package, and it has everything - the CPU/GPU + camera + 4G/3G + wifi, plus you can trivially hook it up to a huge USB powerbank. They even have weatherproofed ones. RPi will cost you more with all the bells and whistles to actually make it work for this case.
- agibsonccc 6y ago
  How did you set it up on the software side though? How flexible/customizable was it?
- stallmanite 6y ago
  I’m really interested in this, do you have anything written about how you did it?
quietbritishjim 6y ago
> You really want to use ... C/C++ for inference, not Python ... > You need ... C/C++, not Python. I think this is a red herring. Usually for deep learning you just use Python to plug together the libraries that actually do the processing, and those are written in terms of C/C++. You can see that in the article where the numpy array returned from OpenCV's video capture API is passed directly to tensorflow. Python never touches the individual pixels of the image directly, and once that's inside tensorflow it's irrelevant that a Python object briefly represented it. > with a Jetson Xavier Well that's obviously the real difference. It's not even just the same general type of computer but a bit faster - the Jetson has a decent NVidia GPU on board whereas the Raspberry Pi is doing the processing on its extremely limp CPU. Indeed that's the whole point of the Jetson; it's basically an NVidia graphics card with extra components strapped to it to turn it into a full computer. > You really want to use DeepStream ... not TensorFlow I'm not familiar with DeepStream, so I'm not so sure about this, but again this is unlikely to make a great deal of difference. It's certainly not the main factor at play here: that's definitely the Jetson's GPU, which of course TensorFlow can certainly use (via CUDA and CUDNN, as does DeepStream). It's true that using TensorRT can provide a speed boost on a Jetson, but even that's possible with TensorFlow, although admittedly you have to remember to call it specifically but it's just three or four lines of (Python!) code. There are already so many ways it's unavoidable to tie yourself into NVidia's ecosystem, it seems like a bad idea to tie yourself in further in a totally avoidable way like this. [Edit: I just realised that the image is being streamed to a remote computer that's doing the inference. The general point remains though. The totally different architecture (including having to transfer data over the network) and hardware are the actual reason for the performance difference, while C/C++ vs Python and DeepStream vs TensorFlow are tiny details.]
- agibsonccc 6y ago
  If something were to be more "neutral" what would you hope to see exactly? Something performant is typically going to be framework/hardware specific.
  - quietbritishjim 6y ago
    Sorry, I'm not sure what you mean by "neutral". Are you talking about my suggestion to avoid DeepStream? If so: The frameworks that work on multiple types of hardware, like TensorFlow and (probably most popular now) PyTorch, have separate backends for their different targets. Each of these backends have huge amounts of platform-specific code, and in the case of the Nvidia backend, that code is written in terms of CUDA just as DeepStream is. That's how they achieve good performance even though the top-level API is hardware generic. The overwhelming majority of deep learning code, both the actual learning and the inference, is written in terms of these frameworks rather than NVidia's proprietary framework. Admittedly I haven't played with NVidia's library, but I highly doubt there's a serious performance difference - it's even possible that the open-source libraries are faster due to the greater community (/Google) effort to optimise them. It does look like DeepStream does a lot more of the processing pipeline than just the inference. In that case it's going to be a lot more tricky to get the whole pipeline on the GPU using those TensorFlow or PyTorch. At the end of the day, if only DeepStream does what you need, I'm not saying you necessarily shouldn't use it - just that you should ideally attempt to avoid it if reasonably possible.
nl 6y ago
Something seems weird here. I agree that Python has some overheard, but the time taken should presumably be dominated by the neural network object detection. In TensorFlow that is written in (highly optimised) C, and should be using the NEON instructions on ARM[1]. Notably, DeepStream gives the same performance with the Python and C++[2]. YOLO inference speed is generally higher than a Mobilenet SSD, but you can run YOLO on TensorFlow instead of Darknet[3], or use a NNPACK version of Darknet. Edit: "I don't need to turn on the noisy Xavier fan." - wait - this isn't on a Raspberry Pi? If you have a GPU on device then there's lots of other things going on. [1] https://www.tensorflow.org/install/source_rpi https://www.tensorflow.org/install/source_rpi [2] https://developer.nvidia.com/deepstream-sdk https://developer.nvidia.com/deepstream-sdk (Scroll down for benchmarks) [3] https://github.com/hunglc007/tensorflow-yolov4-tflite https://github.com/hunglc007/tensorflow-yolov4-tflite
- staycoolboy 6y ago
  > wait - this isn't on a Raspberry Pi? Literally the first sentence of my post. You and many others seem to forget that I explicitly stated I wanted 4 HD streams at 30fps from my home IP address. > but the time taken should presumably be dominated by the neural network object detection There is a lot more to a pipeline than just inference. The problem is aggregating the video streams, downsampling, submitting for inference, and then activating the Gstreamer element to write the MPEG. Most of this can use the nvidia memory properties on the nv* elements, which is great! However eventually you need to copy out for the Arm core. 4 HD streams is a lot of work for a small Arm core. The basic Gstreamer elements do not use the ACL AFAIK. I did recompile them with ORC optimization, but I'm not too familiar how/if that uses ACL/NEON. And the one that I built is basically just a bus messaging system for the downstream codec pipeline that is flagged by the tracker. Can you point me to the benchmark in your link [#2] that indicates Python & C++ have the same performance? Nvidia does have an advantage in that Tensorflow-gpu supports them natively, but that is just for inference. I only see one table and the comments explicitly state they use the DS SDK and -not- TFlow.

exhilaration 6y ago

Do you have any links you could share to build something like this?

reichardt 6y ago

TensorFlow Lite with SSDLite-MobileNet gets you around 4 fps on a Raspberry Pi 4 (23 fps with a Coral USB Accelerator): https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi/blob/master/Raspberry_Pi_Guide.md https://github.com/EdjeElectronics/TensorFlow-Lite-Object-De...

scottlamb 6y ago
You should be able to do a lot better than that if you're careful with the software. As I mentioned in another comment, the Coral USB Accelerator can do at least 100 fps. I haven't looked closely at that link, but likely they're doing H.264 decoding in software using one thread, then downsampling in software using one thread, then waiting for the Coral USB accelerator, and repeating. Maybe they also have the accelerator plugged into a USB 2.0 port rather than a USB 3.0 port. The better approach is to use threading to keep all the Pi's cores busy and the USB accelerator busy at the same time, and to use hardware acceleration.

thebruce87m 6y ago

I’d love to read a write-up of this.

staycoolboy 6y ago
Me too. :) It is literally 90% in the NVIDIA SDK already. The demo examples provided with the kit read an HD stream, run inference and tracking -AND- provide an RTSP output! I started by replacing the HD stream with a videomux from my cameras into a composite image. Working with the the Gstreamer RTSP server is hard, and NVIDIA basically hands it to you. The next steps were to put tee elements on all 4 input streams to a selection demux and recompiled the tracker to send a bus event downstream to my element that controls the output of a video mux. This decides whether to send images to the final "videoconvert ! mp4mux ! payloder ! udpsink" pipeline that goes to a file... sort of. It gets a little messy because I couldn't figure out how to start / stop gst's filesink so I send it to a UDP port instead and have another process on the machine grab payload packets and decide when to create a new file. It used to be one big MP4 file that was pushed to the cloud, and .. um, the program would crash and I would restart the process to get the next file ... i know ... currently I'm trying to chop it up based on idle time (e.g., no new frames in 300ms? start a new file!). It's ugly and I still get corrupt files sometimes, which is why I haven't written it up... and i'm lazy. I bet I could fix this if there was a manual to RTFM, but GStreamer is such a bear to work with and the only source of help is their weird mailing list archive.

scottlamb 6y ago

> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video) I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" because I haven't tested on this exact setup yet. My Macbook Pro and Intel NUC are a lot more pleasant to experiment on (much faster compilation times). A few notes: * I'm currently just using the coral.ai prebuilt 300x300 MobileNet SSD v2 models. I haven't done much testing but can see it has notable false negatives and positives. It'd be wonderful to put together some shared training data [1] to use for transfer learning. I think then results could be much better. Anyone interested in starting something? I'd be happy to contribute! * iirc, I got the Coral USB Accelerator to do about 180 fps with this model. [edit: but don't trust my memory—it could have been as low as 100 fps.] It's easy enough to run the detection at a lower frame rate than the input as well—do the H.264 decoding on every frame but only do inference at fixed pts intervals. * You can also attach multiple Coral USB Accelerators to one system and make use of all of them. * Decoding the 8 streams is likely possible on the Pi 4 depending on your resolution. I haven't messed with this yet, but I think it might even be possible in software, and the Pi has hardware H.264 decoding that I haven't tried to use yet. * I use my cameras' 704x480 "sub" streams for motion detection and downsample that full image to the model's expected 300x300 input. Apparently some people do things like multiple inference against tiles of the image or running a second round of inference against a zoomed-in object detection region to improve confidence. That obviously increases the demand on both the CPU and TPU. * The Orange Pi AI Stick Lite is crazy cheap ($20) and supposedly comparable to the Coral USB Accelerator in speed. At that price if it works buying one per camera doesn't sound too crazy. But I'm not sure if drivers/toolchain support are any good. I have a PLAI Plug (basically the same thing but sold by the manufacturer). The PyTorch-based image classification on a prebuilt model works fine. I don't have the software to build models or do object detection so it's basically useless right now. They want to charge an unknown price for the missing software, but I think Orange Pi's rebrand might include it with the device? [1] https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc/m/rreU83FBBAAJ https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc...

6y ago

haser_au 6y ago

Similar to other's comments, I'd love to read a write up about this.

bufferoverflow 6y ago

I think I was able to run Yolo 3 on my shitty $99 smartphone a while ago. Did it for human detection. Don't remember the FPS, but it wasn't 0.1, it was much better than that. The beauty of a smartphone is it's all in one small package, and it has everything - the CPU/GPU + camera + 4G/3G + wifi, plus you can trivially hook it up to a huge USB powerbank. They even have weatherproofed ones. RPi will cost you more with all the bells and whistles to actually make it work for this case.

agibsonccc 6y ago
How did you set it up on the software side though? How flexible/customizable was it?
stallmanite 6y ago
I’m really interested in this, do you have anything written about how you did it?

quietbritishjim 6y ago

> You really want to use ... C/C++ for inference, not Python ... > You need ... C/C++, not Python. I think this is a red herring. Usually for deep learning you just use Python to plug together the libraries that actually do the processing, and those are written in terms of C/C++. You can see that in the article where the numpy array returned from OpenCV's video capture API is passed directly to tensorflow. Python never touches the individual pixels of the image directly, and once that's inside tensorflow it's irrelevant that a Python object briefly represented it. > with a Jetson Xavier Well that's obviously the real difference. It's not even just the same general type of computer but a bit faster - the Jetson has a decent NVidia GPU on board whereas the Raspberry Pi is doing the processing on its extremely limp CPU. Indeed that's the whole point of the Jetson; it's basically an NVidia graphics card with extra components strapped to it to turn it into a full computer. > You really want to use DeepStream ... not TensorFlow I'm not familiar with DeepStream, so I'm not so sure about this, but again this is unlikely to make a great deal of difference. It's certainly not the main factor at play here: that's definitely the Jetson's GPU, which of course TensorFlow can certainly use (via CUDA and CUDNN, as does DeepStream). It's true that using TensorRT can provide a speed boost on a Jetson, but even that's possible with TensorFlow, although admittedly you have to remember to call it specifically but it's just three or four lines of (Python!) code. There are already so many ways it's unavoidable to tie yourself into NVidia's ecosystem, it seems like a bad idea to tie yourself in further in a totally avoidable way like this. [Edit: I just realised that the image is being streamed to a remote computer that's doing the inference. The general point remains though. The totally different architecture (including having to transfer data over the network) and hardware are the actual reason for the performance difference, while C/C++ vs Python and DeepStream vs TensorFlow are tiny details.]

agibsonccc 6y ago
If something were to be more "neutral" what would you hope to see exactly? Something performant is typically going to be framework/hardware specific.
- quietbritishjim 6y ago
  Sorry, I'm not sure what you mean by "neutral". Are you talking about my suggestion to avoid DeepStream? If so: The frameworks that work on multiple types of hardware, like TensorFlow and (probably most popular now) PyTorch, have separate backends for their different targets. Each of these backends have huge amounts of platform-specific code, and in the case of the Nvidia backend, that code is written in terms of CUDA just as DeepStream is. That's how they achieve good performance even though the top-level API is hardware generic. The overwhelming majority of deep learning code, both the actual learning and the inference, is written in terms of these frameworks rather than NVidia's proprietary framework. Admittedly I haven't played with NVidia's library, but I highly doubt there's a serious performance difference - it's even possible that the open-source libraries are faster due to the greater community (/Google) effort to optimise them. It does look like DeepStream does a lot more of the processing pipeline than just the inference. In that case it's going to be a lot more tricky to get the whole pipeline on the GPU using those TensorFlow or PyTorch. At the end of the day, if only DeepStream does what you need, I'm not saying you necessarily shouldn't use it - just that you should ideally attempt to avoid it if reasonably possible.

nl 6y ago

Something seems weird here. I agree that Python has some overheard, but the time taken should presumably be dominated by the neural network object detection. In TensorFlow that is written in (highly optimised) C, and should be using the NEON instructions on ARM[1]. Notably, DeepStream gives the same performance with the Python and C++[2]. YOLO inference speed is generally higher than a Mobilenet SSD, but you can run YOLO on TensorFlow instead of Darknet[3], or use a NNPACK version of Darknet. Edit: "I don't need to turn on the noisy Xavier fan." - wait - this isn't on a Raspberry Pi? If you have a GPU on device then there's lots of other things going on. [1] https://www.tensorflow.org/install/source_rpi https://www.tensorflow.org/install/source_rpi [2] https://developer.nvidia.com/deepstream-sdk https://developer.nvidia.com/deepstream-sdk (Scroll down for benchmarks) [3] https://github.com/hunglc007/tensorflow-yolov4-tflite https://github.com/hunglc007/tensorflow-yolov4-tflite

staycoolboy 6y ago
> wait - this isn't on a Raspberry Pi? Literally the first sentence of my post. You and many others seem to forget that I explicitly stated I wanted 4 HD streams at 30fps from my home IP address. > but the time taken should presumably be dominated by the neural network object detection There is a lot more to a pipeline than just inference. The problem is aggregating the video streams, downsampling, submitting for inference, and then activating the Gstreamer element to write the MPEG. Most of this can use the nvidia memory properties on the nv* elements, which is great! However eventually you need to copy out for the Arm core. 4 HD streams is a lot of work for a small Arm core. The basic Gstreamer elements do not use the ACL AFAIK. I did recompile them with ORC optimization, but I'm not too familiar how/if that uses ACL/NEON. And the one that I built is basically just a bus messaging system for the downstream codec pipeline that is flagged by the tracker. Can you point me to the benchmark in your link [#2] that indicates Python & C++ have the same performance? Nvidia does have an advantage in that Tensorflow-gpu supports them natively, but that is just for inference. I only see one table and the comments explicitly state they use the DS SDK and -not- TFlow.