4 ms·

Real-time Drone object tracking using Python and OpenCV

110 points by perone 12y ago

hemangshah 12y ago
There are visual tracking techniques that are not only more stable and robust that meanshift, but can also handle occlusions, adapt to changes in appearance models, etc. Have a look at the VOT challenge[1] for the state-of-art in visual tracking. [1] http://www.votchallenge.net/vot2014/download/vot_2014_paper.pdf http://www.votchallenge.net/vot2014/download/vot_2014_paper....
- perone 12y ago
  Thanks for the link, I'm aware of some of the methods cited. This first PoC was just to check how fast and how good Meanshift would work with that image quality. I'll try to implement some more robust approach, specially some method that could handle occlusions.
  - anonymousDan 12y ago
    Out of interest, what kind of hardware specs does the box doing the image processing have (CPU speed, #cores, and memory in particular). Do you expect to be limited by the compute capabilities?
    - zo1 12y ago
      From what I gather, this was running on the drone itself, in real time. #Edit. Scratch that, the code is opening a file "upabove.mp4"... i.e. not realtime, on the drone. Realtime on a PC/laptop.
      - perone 12y ago
        You're correct, this is a post-processing operation, however I took care to do something that works in real time, so you can use this same algorithm for any FPV transmission, which is the next step that I'm planning to implement (but with more robust features).
  - hemangshah 12y ago
    Christian, please also be aware that some of the techniques do not incorporate changes in scale of the tracked object. If you are working with aerial footage then this might be an important consideration. I've recently worked on a similar problem, so you need any help, please feel free to email me (email address is given in my profile)
    - perone 12y ago
      Thanks for the feedback ! I'm planning to use more robust and scale invariant features for this so this problem of different levels of zoom would be solved (at least to the limits of what can be done).
adriancooney 12y ago
This isn't exactly "real time" since he loads an MP4 file (`cv2.VideoCapture('upabove.mp4')`). I think it would be a lot more impressive if the drone was able to stream the film to a server of some kind, process it there then make decisions on it's flight path.
- dharma1 12y ago
  Agree the headline makes it sounds like it was done live. In fact I don't see why the drone bit is relevant here, any shaky footage would have done. Some more challenges involved with using live footage to control a drone (using a companion computer and talk to APM using mavlink to control the drone realtime)
- infinitone 12y ago
  That's what I was thinking too. Also, isn't this sort of thing pretty common now? As in, aren't there startups that have drones that follow you to take cool video of you [1]. I'd imagine they are doing some sort of realtime tracking. [1] https://www.airdog.com/ https://www.airdog.com/
  - andymoe 12y ago
    None of the "follow me" drones folks are really doing object tracking yet. They basically just "fake it" by sending a bunch of guided waypoints with the offset of the phone or bracelet GPS location in airdogs case and hope it looks OK because of the really wide angle lens of the GoPro. There is a lot of room for improvement in this area that could be made by feeding that realtime tracking data back into the flight controller as you suggest. The one startup in the drone space that is really putting computer vision stuff to work on drones is http://skyd.io http://skyd.io - some ex google project wing folks. The demos are freaking amazing. Wait long enough on the website background video and you can see a six foot wingspan UAV flying itself through a parking garage. Source: I live an breath drone stuff and write a UAV ground station app for iOS to go along with our Apple MFi certified hardware for the long range telemetry connection from iOS. It works with OPs flight controller by the way :-)
PanMan 12y ago
This works better than I expected, but I'm wondering how well it will work if the angle changes (flying around the building) or other parameters change. But looks impressive for how little code it is!
- zo1 12y ago
  If the angle changes, the details of the building's facade changes as well. You can see in the code that he isolates the building's details from the first frame. Then compares later images to it to do the "tracking".

110 points by perone 12y ago

There are visual tracking techniques that are not only more stable and robust that meanshift, but can also handle occlusions, adapt to changes in appearance models, etc. Have a look at the VOT challenge[1] for the state-of-art in visual tracking. [1] http://www.votchallenge.net/vot2014/download/vot_2014_paper.pdf http://www.votchallenge.net/vot2014/download/vot_2014_paper....

perone 12y ago
Thanks for the link, I'm aware of some of the methods cited. This first PoC was just to check how fast and how good Meanshift would work with that image quality. I'll try to implement some more robust approach, specially some method that could handle occlusions.
- anonymousDan 12y ago
  Out of interest, what kind of hardware specs does the box doing the image processing have (CPU speed, #cores, and memory in particular). Do you expect to be limited by the compute capabilities?
  - zo1 12y ago
    From what I gather, this was running on the drone itself, in real time. #Edit. Scratch that, the code is opening a file "upabove.mp4"... i.e. not realtime, on the drone. Realtime on a PC/laptop.
    - perone 12y ago
      You're correct, this is a post-processing operation, however I took care to do something that works in real time, so you can use this same algorithm for any FPV transmission, which is the next step that I'm planning to implement (but with more robust features).
- hemangshah 12y ago
  Christian, please also be aware that some of the techniques do not incorporate changes in scale of the tracked object. If you are working with aerial footage then this might be an important consideration. I've recently worked on a similar problem, so you need any help, please feel free to email me (email address is given in my profile)
  - perone 12y ago
    Thanks for the feedback ! I'm planning to use more robust and scale invariant features for this so this problem of different levels of zoom would be solved (at least to the limits of what can be done).

adriancooney 12y ago

This isn't exactly "real time" since he loads an MP4 file (`cv2.VideoCapture('upabove.mp4')`). I think it would be a lot more impressive if the drone was able to stream the film to a server of some kind, process it there then make decisions on it's flight path.

dharma1 12y ago
Agree the headline makes it sounds like it was done live. In fact I don't see why the drone bit is relevant here, any shaky footage would have done. Some more challenges involved with using live footage to control a drone (using a companion computer and talk to APM using mavlink to control the drone realtime)
infinitone 12y ago
That's what I was thinking too. Also, isn't this sort of thing pretty common now? As in, aren't there startups that have drones that follow you to take cool video of you [1]. I'd imagine they are doing some sort of realtime tracking. [1] https://www.airdog.com/ https://www.airdog.com/
- andymoe 12y ago
  None of the "follow me" drones folks are really doing object tracking yet. They basically just "fake it" by sending a bunch of guided waypoints with the offset of the phone or bracelet GPS location in airdogs case and hope it looks OK because of the really wide angle lens of the GoPro. There is a lot of room for improvement in this area that could be made by feeding that realtime tracking data back into the flight controller as you suggest. The one startup in the drone space that is really putting computer vision stuff to work on drones is http://skyd.io http://skyd.io - some ex google project wing folks. The demos are freaking amazing. Wait long enough on the website background video and you can see a six foot wingspan UAV flying itself through a parking garage. Source: I live an breath drone stuff and write a UAV ground station app for iOS to go along with our Apple MFi certified hardware for the long range telemetry connection from iOS. It works with OPs flight controller by the way :-)

PanMan 12y ago

This works better than I expected, but I'm wondering how well it will work if the angle changes (flying around the building) or other parameters change. But looks impressive for how little code it is!

zo1 12y ago
If the angle changes, the details of the building's facade changes as well. You can see in the code that he isolates the building's details from the first frame. Then compares later images to it to do the "tracking".