5 ms·

This isn't exactly "real time" since he loads an MP4 file (`cv2.VideoCapture('upabove.mp4')`). I think it would be a lot more impressive if the drone was able t

This isn't exactly "real time" since he loads an MP4 file (`cv2.VideoCapture('upabove.mp4')`). I think it would be a lot more impressive if the drone was able to stream the film to a server of some kind, process it there then make decisions on it's flight path.

dharma1 12y ago
Agree the headline makes it sounds like it was done live. In fact I don't see why the drone bit is relevant here, any shaky footage would have done. Some more challenges involved with using live footage to control a drone (using a companion computer and talk to APM using mavlink to control the drone realtime)
infinitone 12y ago
That's what I was thinking too. Also, isn't this sort of thing pretty common now? As in, aren't there startups that have drones that follow you to take cool video of you [1]. I'd imagine they are doing some sort of realtime tracking. [1] https://www.airdog.com/ https://www.airdog.com/
- andymoe 12y ago
  None of the "follow me" drones folks are really doing object tracking yet. They basically just "fake it" by sending a bunch of guided waypoints with the offset of the phone or bracelet GPS location in airdogs case and hope it looks OK because of the really wide angle lens of the GoPro. There is a lot of room for improvement in this area that could be made by feeding that realtime tracking data back into the flight controller as you suggest. The one startup in the drone space that is really putting computer vision stuff to work on drones is http://skyd.io http://skyd.io - some ex google project wing folks. The demos are freaking amazing. Wait long enough on the website background video and you can see a six foot wingspan UAV flying itself through a parking garage. Source: I live an breath drone stuff and write a UAV ground station app for iOS to go along with our Apple MFi certified hardware for the long range telemetry connection from iOS. It works with OPs flight controller by the way :-)
  - dharma1 12y ago
    OP's video doesn't really have anything to do with drones as such, just loads an mp4 of drone footage. The fact that it's drone footage is completely incidental. But lots of people are doing real computer vision and object tracking on drones and robots, and have been doing for years. https://www.youtube.com/watch?v=Gj-5RNdUz3I https://www.youtube.com/watch?v=Gj-5RNdUz3I https://www.youtube.com/watch?v=HqzMSxBOV-U https://www.youtube.com/watch?v=HqzMSxBOV-U https://www.youtube.com/watch?v=C95bngCOv9Q https://www.youtube.com/watch?v=C95bngCOv9Q
    - andymoe 12y ago
      I know. I was specifically addressing the "follow me" class of drones mentioned that are getting a lot of press these days such as airdog, hexoplus, 3dr IRIS+ (all APM:Copter based) and no one outside university or military settings that I know of have really integrated that data back into the flight controller as well as the skydio demo suggests.
      - votingprawn 12y ago
        Unless they've had a massive and sudden change of heart, Airdog is based on the PX4/NuttX stack, not on the APM:Copter.
        andymoe 12y ago
        Sorry, it gets confusing since APM:Copter autopilot (I guess that's what they are calling it now) can run on PX4 hardware running NuttX too. So they are using the full PX4 flight stack as well?
        votingprawn 12y ago
        Yep, although as I understand it they've worked so hard on it since the Kickstarter I'm not sure how much is still recognisable as the core ETH code. At the very least their software is rooted firmly in the PX4/Lorenz design ethos. One of the guys behind the project is also the author of some very nice PX4 and MAVLink tools which you may well have used: https://github.com/DrTon https://github.com/DrTon [we have a nightmare at work trying to distinguish between original PX4FMU running PX4, Pixhawks running PX4, and Pixhawks running APM codebase in conversations]
lovelearning 12y ago
Few days ago, there was a post here on HN on an add-on board that lets a raspberry pi control a copter[1]. I think your idea would be a very good use case for it, because it's capable of running opencv, can control camera position, and avoids downlink latency since all processing is happening on board. [1]: https://news.ycombinator.com/item?id=8874144 https://news.ycombinator.com/item?id=8874144
perone 12y ago
You're correct, however I took care to make the algorithm work in realtime, so you can use it in any FPV transmission. I'm going to prepare a good FPV system to run this with video transmission, but since the algorithm is very fast, it can be executed by an Raspberry Pi or a Beagleboard for instance (embedded in the drone), to avoid loss of quality of the transmission and issues like that.

dharma1 12y ago

Agree the headline makes it sounds like it was done live. In fact I don't see why the drone bit is relevant here, any shaky footage would have done. Some more challenges involved with using live footage to control a drone (using a companion computer and talk to APM using mavlink to control the drone realtime)

infinitone 12y ago

That's what I was thinking too. Also, isn't this sort of thing pretty common now? As in, aren't there startups that have drones that follow you to take cool video of you [1]. I'd imagine they are doing some sort of realtime tracking. [1] https://www.airdog.com/ https://www.airdog.com/

andymoe 12y ago
None of the "follow me" drones folks are really doing object tracking yet. They basically just "fake it" by sending a bunch of guided waypoints with the offset of the phone or bracelet GPS location in airdogs case and hope it looks OK because of the really wide angle lens of the GoPro. There is a lot of room for improvement in this area that could be made by feeding that realtime tracking data back into the flight controller as you suggest. The one startup in the drone space that is really putting computer vision stuff to work on drones is http://skyd.io http://skyd.io - some ex google project wing folks. The demos are freaking amazing. Wait long enough on the website background video and you can see a six foot wingspan UAV flying itself through a parking garage. Source: I live an breath drone stuff and write a UAV ground station app for iOS to go along with our Apple MFi certified hardware for the long range telemetry connection from iOS. It works with OPs flight controller by the way :-)
- dharma1 12y ago
  OP's video doesn't really have anything to do with drones as such, just loads an mp4 of drone footage. The fact that it's drone footage is completely incidental. But lots of people are doing real computer vision and object tracking on drones and robots, and have been doing for years. https://www.youtube.com/watch?v=Gj-5RNdUz3I https://www.youtube.com/watch?v=Gj-5RNdUz3I https://www.youtube.com/watch?v=HqzMSxBOV-U https://www.youtube.com/watch?v=HqzMSxBOV-U https://www.youtube.com/watch?v=C95bngCOv9Q https://www.youtube.com/watch?v=C95bngCOv9Q
  - andymoe 12y ago
    I know. I was specifically addressing the "follow me" class of drones mentioned that are getting a lot of press these days such as airdog, hexoplus, 3dr IRIS+ (all APM:Copter based) and no one outside university or military settings that I know of have really integrated that data back into the flight controller as well as the skydio demo suggests.
    - votingprawn 12y ago
      Unless they've had a massive and sudden change of heart, Airdog is based on the PX4/NuttX stack, not on the APM:Copter.
      - andymoe 12y ago
        Sorry, it gets confusing since APM:Copter autopilot (I guess that's what they are calling it now) can run on PX4 hardware running NuttX too. So they are using the full PX4 flight stack as well?
        votingprawn 12y ago
        Yep, although as I understand it they've worked so hard on it since the Kickstarter I'm not sure how much is still recognisable as the core ETH code. At the very least their software is rooted firmly in the PX4/Lorenz design ethos. One of the guys behind the project is also the author of some very nice PX4 and MAVLink tools which you may well have used: https://github.com/DrTon https://github.com/DrTon [we have a nightmare at work trying to distinguish between original PX4FMU running PX4, Pixhawks running PX4, and Pixhawks running APM codebase in conversations]

lovelearning 12y ago

Few days ago, there was a post here on HN on an add-on board that lets a raspberry pi control a copter[1]. I think your idea would be a very good use case for it, because it's capable of running opencv, can control camera position, and avoids downlink latency since all processing is happening on board. [1]: https://news.ycombinator.com/item?id=8874144 https://news.ycombinator.com/item?id=8874144

perone 12y ago

You're correct, however I took care to make the algorithm work in realtime, so you can use it in any FPV transmission. I'm going to prepare a good FPV system to run this with video transmission, but since the algorithm is very fast, it can be executed by an Raspberry Pi or a Beagleboard for instance (embedded in the drone), to avoid loss of quality of the transmission and issues like that.