7 ms·

Flying a Drone in First Person View Using the Oculus Rift

81 points by radley 13y ago

andymoe 13y ago
This is really cool. A few buddies and myself got this setup working with an AR drone and a Rift a few months back. I'll try to get videos up somewhere soon. If you are into this kind of stuff and in the Bay Area check out this meetup: http://www.meetup.com/SF-Drones-Startup-Meetup http://www.meetup.com/SF-Drones-Startup-Meetup We had the people from Velodyne who make the 3d Lidar for googles self driving cars present last night and demoed their newer 1kg unit.
darkmighty 13y ago
People having been flying with a FPV headset on RC planes for a while, but hopefully the Rift will be a cheaper and better quality option. Also, this makes it easier to make a rotating camera using the Rift's integrated IMU, for that truly immersive experience!
- andymoe 13y ago
  I'll second this. The IMU (And Rift SDK) is really really good. I can't wait to get my hands on the HD version of Rift.
danieldrehmer 13y ago
What is the point of this? The cool thing about the rift is how modern components and graphics capabilities allow for a light VR device to provide a very immersive experience. This is basically being used to show video from the drone's camera. No stereoscopic image, no controlling using the gyro/accelerometers. Also, low latency, that is one of the most important aspects of what makes the rift experience interesting, certainly wasn't part of this experiment. No point in posting a video showing new/trendy devices being used in such a meaningless way, just for the sake of combining cool toys
- Laremere 13y ago
  > No stereoscopic image Did you even click the link? Of the three sentences, the second one starts out with "Using its two cameras", and the two cameras are shown within the first 10 seconds of video. >What is the point of this? The stereo view allows for depth perception in flight. They're getting fish eye lenses, and one thing that's hard to express about VR helmets like this one is the sense of space around you, even without stereo vision. This will create a feeling of flying that no monitor could ever hope to achieve.
- deleted 13y ago
  [deleted]
fsckin 13y ago
Oculus is perfect for this. I was really hoping the cameras would be mounted on a 3-axis gimbal, with each axis controlled by the Oculus motion sensing. The Oculus is a great option for for first person flying, even with using it as a regular 2D display, $300 is pretty cheap compared to most head mounted displays.
- tlrobinson 13y ago
  The latency of physically moving cameras with head tracking would be horrendous. Perhaps if you also used wide-angle/fisheye lenses and somehow synchronized the virtual panning with physical panning it would be workable, as long as you don't move your head too quickly.
  - mwilcox 13y ago
    The control doesn't have to be 100% absolute. You can have the video 'projected' within a 3D environment and adjust it's position within that environment based on the camera's physical movements, but you can let the 1:1 tracking from the headset overshoot the boundaries so the user doesn't get too much of a jarring experience. It would feel as if they were in a cockpit.
    - anigbrowl 13y ago
      Trying that approach will probably result in people throwing up. In an immersive environment even short latencies are extremely disorienting. When I first started recording audio for film, the industry-standard recorders used Digital Audio Tape and there was a switch to monitor either the live feed from the preamps or the recorded feed from the tape (so you could be sure you were recording - you'd be surprised how easy it is to mess this up over the course of a 12-hour workday). The latency was small, on the order of 10-12 milliseconds, but it took me months to overcome the weirdness of opening your mouth to speak but not hearing anything until ~1/100th of a second later. I shudder to think what it would be like to have this in your visual cortex but with a longer delay and also while your eyes are telling you that you're floating above the ground.
DanielBMarkham 13y ago
For those of us with 3-D stereographic displays or TVs, anybody know if there's an mpo file of this test?
sytelus 13y ago
Does anyone know which RC drone are they using for this?
- andymoe 13y ago
  I think this is the drone they are flying with... http://blackarmoreddrone.com/store/ http://blackarmoreddrone.com/store/ Probably just a touch out of the hobby market price range with a $49k base. I think that's US dollars but I'm not 100pct sure.
  - yial 13y ago
    They're based in Sweden I believe, so even if it was in Krona it would still be around $7,500 USD, which is still pretty pricey. Though based on their website (which doesn't appear to use geo location by ip) I think it's USD as well.
olex 13y ago
This is cool, but the Oculus is not the optimal tool for the job right now. I'm in the FPV hobby for over half a year and fly quadrocopters and model airplanes using video goggles. The Rift currently has one important problem that, until overcome, will prevent it from gaining traction in this area: latency. It doesn't have an analog video input, and having to digitalize and process the video for the Rift introduces a delay into the signal that makes controlling your model uncomfortable at best, impossible in many cases. Because of this problem, pretty much everybody uses analog video for FPV these days. There are goggles on the market made specifically for the job: FatShark series, the Zeiss Cinemizer HD and others. Some of them are HD and 3D capable (Cinemizers), many are fitted with built-in motion trackers (e.g. FatShark Attidude SD) that are easily bound to servos on the model controlling camera movement (pan-tilt setups), and there are external motion tracker modules easily used with the goggles that don't come with them. With no delay, flying even very fast aircraft is actually much easier in FPV than using traditional line-of-sight control. The latency problem is worse than it might sound: e.g. the GoPro cameras introduce a very small delay on their analog outputs (<50ms), and even this is absolutely noticeable by an experienced pilot in FPV flight and influences the control the pilot has considerably; the best latency currently achieved with various FPV setups using the Oculus Rift is in the 200ms region. Another advantage of using "traditional" motion trackers over the Oculus Rift is that they output PWM signal, ready to be fed into your R/C transmitter and then used on the model with servos to move the camera. Now, I understand that HD and 3D is eventually the future of this hobby, and the Oculus Rift might very well at some point become a very viable goggles and motion tracker assembly. But until the latency issue is solved, it's not going to gain a wide audience.
- DigitalJack 13y ago
  I used to fly RC airplanes back when I was in high school. I've been out of the loop for 20 years now... are there good sites to re-engage the RC world and/or get into FPV?
  - commandar 13y ago
    I recently got into the hobby and found that Youtube was honestly the best source of information. There's a channel called RC Model Reviews that's particularly good. It's run by a guy named Bruce Simpson that's a bit of a crazy old Kiwi but he has a ton of really solid explanations of nearly everything you need to know getting started out. One cool thing is that the past couple of years have seen some serious disruption in the field. Open source firmwares for controllers are a thing now, and it's had nearly the same effect on RC that the WRT54G had on the consumer router business - you can now do things with very cheap hardware that used to cost hundreds or thousands of dollars. Some links for you: ER9X Firmware - https://code.google.com/p/er9x/ https://code.google.com/p/er9x/ FrSky, a company doing a lot of interesting things in terms of radio equipment - http://www.frsky-rc.com/ http://www.frsky-rc.com/ The aforementioned RC Model Reviews - http://www.youtube.com/user/RCModelReviews http://www.youtube.com/user/RCModelReviews
marcosscriven 13y ago
Looks like fun. A shame the left/right views aren't on opposite sides, then one could view it cross-eyed to get the 3D effect without glasses. I wonder if it would make you nauseous looking through the Occulus?
phryk 13y ago
This really doesn't seem impressive. Why the heck do they even make the copter fly around with a WHOLE GODDAMN LAPTOP, this is about as wasteful as you can possibly get… Something better which I've already seen is a gimbal[1] (which you can build for <100€) with one or two (for 3D) gopro cams. Weighs less and is way more awesome. [1] http://s3files.core77.com/blog/images/2013/04/freefly-systems-movi-02.gif http://s3files.core77.com/blog/images/2013/04/freefly-system... Addendum: This is the sort of steadiness you get with a gimbal btw: https://www.youtube.com/watch?v=NmSFZydbVnc https://www.youtube.com/watch?v=NmSFZydbVnc

81 points by radley 13y ago

This is really cool. A few buddies and myself got this setup working with an AR drone and a Rift a few months back. I'll try to get videos up somewhere soon. If you are into this kind of stuff and in the Bay Area check out this meetup: http://www.meetup.com/SF-Drones-Startup-Meetup http://www.meetup.com/SF-Drones-Startup-Meetup We had the people from Velodyne who make the 3d Lidar for googles self driving cars present last night and demoed their newer 1kg unit.

darkmighty 13y ago

People having been flying with a FPV headset on RC planes for a while, but hopefully the Rift will be a cheaper and better quality option. Also, this makes it easier to make a rotating camera using the Rift's integrated IMU, for that truly immersive experience!

andymoe 13y ago
I'll second this. The IMU (And Rift SDK) is really really good. I can't wait to get my hands on the HD version of Rift.

danieldrehmer 13y ago

What is the point of this? The cool thing about the rift is how modern components and graphics capabilities allow for a light VR device to provide a very immersive experience. This is basically being used to show video from the drone's camera. No stereoscopic image, no controlling using the gyro/accelerometers. Also, low latency, that is one of the most important aspects of what makes the rift experience interesting, certainly wasn't part of this experiment. No point in posting a video showing new/trendy devices being used in such a meaningless way, just for the sake of combining cool toys

Laremere 13y ago
> No stereoscopic image Did you even click the link? Of the three sentences, the second one starts out with "Using its two cameras", and the two cameras are shown within the first 10 seconds of video. >What is the point of this? The stereo view allows for depth perception in flight. They're getting fish eye lenses, and one thing that's hard to express about VR helmets like this one is the sense of space around you, even without stereo vision. This will create a feeling of flying that no monitor could ever hope to achieve.
deleted 13y ago
[deleted]

fsckin 13y ago

Oculus is perfect for this. I was really hoping the cameras would be mounted on a 3-axis gimbal, with each axis controlled by the Oculus motion sensing. The Oculus is a great option for for first person flying, even with using it as a regular 2D display, $300 is pretty cheap compared to most head mounted displays.

tlrobinson 13y ago
The latency of physically moving cameras with head tracking would be horrendous. Perhaps if you also used wide-angle/fisheye lenses and somehow synchronized the virtual panning with physical panning it would be workable, as long as you don't move your head too quickly.
- mwilcox 13y ago
  The control doesn't have to be 100% absolute. You can have the video 'projected' within a 3D environment and adjust it's position within that environment based on the camera's physical movements, but you can let the 1:1 tracking from the headset overshoot the boundaries so the user doesn't get too much of a jarring experience. It would feel as if they were in a cockpit.
  - anigbrowl 13y ago
    Trying that approach will probably result in people throwing up. In an immersive environment even short latencies are extremely disorienting. When I first started recording audio for film, the industry-standard recorders used Digital Audio Tape and there was a switch to monitor either the live feed from the preamps or the recorded feed from the tape (so you could be sure you were recording - you'd be surprised how easy it is to mess this up over the course of a 12-hour workday). The latency was small, on the order of 10-12 milliseconds, but it took me months to overcome the weirdness of opening your mouth to speak but not hearing anything until ~1/100th of a second later. I shudder to think what it would be like to have this in your visual cortex but with a longer delay and also while your eyes are telling you that you're floating above the ground.

DanielBMarkham 13y ago

For those of us with 3-D stereographic displays or TVs, anybody know if there's an mpo file of this test?

sytelus 13y ago

Does anyone know which RC drone are they using for this?

andymoe 13y ago
I think this is the drone they are flying with... http://blackarmoreddrone.com/store/ http://blackarmoreddrone.com/store/ Probably just a touch out of the hobby market price range with a $49k base. I think that's US dollars but I'm not 100pct sure.
- yial 13y ago
  They're based in Sweden I believe, so even if it was in Krona it would still be around $7,500 USD, which is still pretty pricey. Though based on their website (which doesn't appear to use geo location by ip) I think it's USD as well.

olex 13y ago

This is cool, but the Oculus is not the optimal tool for the job right now. I'm in the FPV hobby for over half a year and fly quadrocopters and model airplanes using video goggles. The Rift currently has one important problem that, until overcome, will prevent it from gaining traction in this area: latency. It doesn't have an analog video input, and having to digitalize and process the video for the Rift introduces a delay into the signal that makes controlling your model uncomfortable at best, impossible in many cases. Because of this problem, pretty much everybody uses analog video for FPV these days. There are goggles on the market made specifically for the job: FatShark series, the Zeiss Cinemizer HD and others. Some of them are HD and 3D capable (Cinemizers), many are fitted with built-in motion trackers (e.g. FatShark Attidude SD) that are easily bound to servos on the model controlling camera movement (pan-tilt setups), and there are external motion tracker modules easily used with the goggles that don't come with them. With no delay, flying even very fast aircraft is actually much easier in FPV than using traditional line-of-sight control. The latency problem is worse than it might sound: e.g. the GoPro cameras introduce a very small delay on their analog outputs (<50ms), and even this is absolutely noticeable by an experienced pilot in FPV flight and influences the control the pilot has considerably; the best latency currently achieved with various FPV setups using the Oculus Rift is in the 200ms region. Another advantage of using "traditional" motion trackers over the Oculus Rift is that they output PWM signal, ready to be fed into your R/C transmitter and then used on the model with servos to move the camera. Now, I understand that HD and 3D is eventually the future of this hobby, and the Oculus Rift might very well at some point become a very viable goggles and motion tracker assembly. But until the latency issue is solved, it's not going to gain a wide audience.

DigitalJack 13y ago
I used to fly RC airplanes back when I was in high school. I've been out of the loop for 20 years now... are there good sites to re-engage the RC world and/or get into FPV?
- commandar 13y ago
  I recently got into the hobby and found that Youtube was honestly the best source of information. There's a channel called RC Model Reviews that's particularly good. It's run by a guy named Bruce Simpson that's a bit of a crazy old Kiwi but he has a ton of really solid explanations of nearly everything you need to know getting started out. One cool thing is that the past couple of years have seen some serious disruption in the field. Open source firmwares for controllers are a thing now, and it's had nearly the same effect on RC that the WRT54G had on the consumer router business - you can now do things with very cheap hardware that used to cost hundreds or thousands of dollars. Some links for you: ER9X Firmware - https://code.google.com/p/er9x/ https://code.google.com/p/er9x/ FrSky, a company doing a lot of interesting things in terms of radio equipment - http://www.frsky-rc.com/ http://www.frsky-rc.com/ The aforementioned RC Model Reviews - http://www.youtube.com/user/RCModelReviews http://www.youtube.com/user/RCModelReviews

marcosscriven 13y ago

Looks like fun. A shame the left/right views aren't on opposite sides, then one could view it cross-eyed to get the 3D effect without glasses. I wonder if it would make you nauseous looking through the Occulus?

phryk 13y ago

This really doesn't seem impressive. Why the heck do they even make the copter fly around with a WHOLE GODDAMN LAPTOP, this is about as wasteful as you can possibly get… Something better which I've already seen is a gimbal[1] (which you can build for <100€) with one or two (for 3D) gopro cams. Weighs less and is way more awesome. [1] http://s3files.core77.com/blog/images/2013/04/freefly-systems-movi-02.gif http://s3files.core77.com/blog/images/2013/04/freefly-system... Addendum: This is the sort of steadiness you get with a gimbal btw: https://www.youtube.com/watch?v=NmSFZydbVnc https://www.youtube.com/watch?v=NmSFZydbVnc