4 ms·
> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow onlin
by scottlamb 6y ago
> You need way more processing power than an RPi to do this at 30fps, and C/C++, not Python. (There are literally dozens of projects for the RPi and TFlow online but they all get like 0.1 fps or less by using Flask and browser reload of a PNG... great for POC but not for real video)
I think 8 streams at 15 fps (aka 120 fps total) is possible with a ($35) Raspberry Pi 4 + ($75) Coral USB Accelerator. I say "I think" because I haven't tested on this exact setup yet. My Macbook Pro and Intel NUC are a lot more pleasant to experiment on (much faster compilation times). A few notes:
* I'm currently just using the coral.ai prebuilt 300x300 MobileNet SSD v2 models. I haven't done much testing but can see it has notable false negatives and positives. It'd be wonderful to put together some shared training data [1] to use for transfer learning. I think then results could be much better. Anyone interested in starting something? I'd be happy to contribute!
* iirc, I got the Coral USB Accelerator to do about 180 fps with this model. [edit: but don't trust my memory—it could have been as low as 100 fps.] It's easy enough to run the detection at a lower frame rate than the input as well—do the H.264 decoding on every frame but only do inference at fixed pts intervals.
* You can also attach multiple Coral USB Accelerators to one system and make use of all of them.
* Decoding the 8 streams is likely possible on the Pi 4 depending on your resolution. I haven't messed with this yet, but I think it might even be possible in software, and the Pi has hardware H.264 decoding that I haven't tried to use yet.
* I use my cameras' 704x480 "sub" streams for motion detection and downsample that full image to the model's expected 300x300 input. Apparently some people do things like multiple inference against tiles of the image or running a second round of inference against a zoomed-in object detection region to improve confidence. That obviously increases the demand on both the CPU and TPU.
* The Orange Pi AI Stick Lite is crazy cheap ($20) and supposedly comparable to the Coral USB Accelerator in speed. At that price if it works buying one per camera doesn't sound too crazy. But I'm not sure if drivers/toolchain support are any good. I have a PLAI Plug (basically the same thing but sold by the manufacturer). The PyTorch-based image classification on a prebuilt model works fine. I don't have the software to build models or do object detection so it's basically useless right now. They want to charge an unknown price for the missing software, but I think Orange Pi's rebrand might include it with the device?
[1] https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc/m/rreU83FBBAAJ https://groups.google.com/g/moonfire-nvr-users/c/ZD1uS7kL7tc...
- serf 6y ago>* I use my cameras' 704x480 "sub" streams for motion detection and downsample.. i've encountered cheap IPTV cameras where the main high-res stream was actually being offered with a time-shift compared to the sub-stream. weird shit happens when you have a camera that does that, then you act on data from the sub-stream to work with data on the main stream. I played with a 'Chinesium' cctv with generic firmware that had such a bad offset that I could actually use a static offset to remediate it. I assumed it was just a firmware bug, since the offsets didn't seem to move around as if it was a decode/encode lag or anything of that sort.
- scottlamb 6y agoYeah, that sucks. Did the camera send SEI Picture Timing messages? RTCP Sender Reports with NTP timestamps? Either could potentially help matters if they're trustworthy. I haven't encountered that exact problem (large fixed offset between the streams), but I agree in general these cameras' time support is poor and synchronizing streams (either between main/sub of a single camera or across cameras) is a pain point. Here's what my software is doing today: https://github.com/scottlamb/moonfire-nvr/blob/master/design/time.md https://github.com/scottlamb/moonfire-nvr/blob/master/design... Any of several changes to the camera would improve matters a lot: * using temporal/spatial/quality SVC (Scalable Video Coding) so you can get everything you need from a single video stream * exposing timestamps relative to the camera's uptime (CLOCK_MONOTONIC) somehow (not sure where you'd cram this into a RTSP session) along with some random boot id * allow fetching both the main and sub video streams in a single RTSP session * reliably slewing the clock like a "real" NTP client rather than stepping with SNTP but I'm not exactly in a position to make suggestions that the camera manufacturers jump to implement...
- nl 6y agoThe Orange Pi AI Stick Lite looks really interesting. Here's the link: https://www.aliexpress.com/item/32958159325.html https://www.aliexpress.com/item/32958159325.html and it says the PLAI training tools are (now?) free on request.
- scottlamb 6y ago