3 ms·

I hit upon a similar idea back in about 2000... the key at the time was video encoding hardware that was designed to incorporate MPEG compression. Pulling motio

I hit upon a similar idea back in about 2000... the key at the time was video encoding hardware that was designed to incorporate MPEG compression. Pulling motion vectors from the encoded stream could have given realtime optical flow way back then. Unfortunately I was nowhere good enough as a programmer to take advantage of it for real projects. A team at MIT had a paper about this that I was able to find. Had a page about it on robots.net, but it seems to be down now

tbirdz 8y ago
We're doing this with H.264 at work. Pretty useful since we need to generate the encoded stream anyway, so now we get motion estimation for free.
- rjeli 8y ago
  Awesome. Which encoder provides this? Can you pull it from cuvid?
  - tbirdz 8y ago
    In our case it's a hardware h264 encoder in an embedded product, so our method wouldn't be any of any use for you in a more general case. We sussed out an undocumented debug api to get the chip to dma the motion vectors to a debug buffer, and then we can extract and use them. I think those apis are originally intended to be used if a customer is having issues with the product so that the vendor engineers can help troubleshoot. We were originally looking at analyzing the h264 bitstream in software and getting the vectors out that way. Honestly I don't think that would be extraordinarily difficult, there's already code in ffmpeg to draw the motion vectors as arrows on the decoded frame, so that would be a good place to look if there aren't any preexisting apis or tools to extract them. Good luck!
  - haxiomic 7y ago
    NVidia has some example code for getting the motion vectors during video encoding https://github.com/NVIDIA/video-sdk-samples/blob/master/Samples/AppEncode/AppEncME/AppEncME.cpp https://github.com/NVIDIA/video-sdk-samples/blob/master/Samp...
    - rjeli 7y ago
      Thank you!
stefan_ 8y ago
The H.264 hardware encoder in the Raspberry Pi can provide the motion vectors: https://picamera.readthedocs.io/en/release-1.13/recipes2.html#recording-motion-vector-data https://picamera.readthedocs.io/en/release-1.13/recipes2.htm...
deleted 8y ago
[deleted]

tbirdz 8y ago

We're doing this with H.264 at work. Pretty useful since we need to generate the encoded stream anyway, so now we get motion estimation for free.

rjeli 8y ago
Awesome. Which encoder provides this? Can you pull it from cuvid?
- tbirdz 8y ago
  In our case it's a hardware h264 encoder in an embedded product, so our method wouldn't be any of any use for you in a more general case. We sussed out an undocumented debug api to get the chip to dma the motion vectors to a debug buffer, and then we can extract and use them. I think those apis are originally intended to be used if a customer is having issues with the product so that the vendor engineers can help troubleshoot. We were originally looking at analyzing the h264 bitstream in software and getting the vectors out that way. Honestly I don't think that would be extraordinarily difficult, there's already code in ffmpeg to draw the motion vectors as arrows on the decoded frame, so that would be a good place to look if there aren't any preexisting apis or tools to extract them. Good luck!
- haxiomic 7y ago
  NVidia has some example code for getting the motion vectors during video encoding https://github.com/NVIDIA/video-sdk-samples/blob/master/Samples/AppEncode/AppEncME/AppEncME.cpp https://github.com/NVIDIA/video-sdk-samples/blob/master/Samp...
  - rjeli 7y ago
    Thank you!

stefan_ 8y ago

The H.264 hardware encoder in the Raspberry Pi can provide the motion vectors: https://picamera.readthedocs.io/en/release-1.13/recipes2.html#recording-motion-vector-data https://picamera.readthedocs.io/en/release-1.13/recipes2.htm...

deleted 8y ago

[deleted]