5 ms·

This is one of those things I'm betting that after a few years and some simplifications, every new high-end smartphone will have as a standard feature.

GrantS 13y ago
So this is a topic I have thought about for awhile -- I released a 3D scanning app for iPhone nearly 3 years ago (uses the screen as the light source for scanning, called Trimensional) and have had many conversations about when Kinect-style sensing would become standard. If we just take Apple as an example, for them to include it standard, it would have to really justify its existence. (I realize many people don't care about Apple, but I choose Apple because, for example, there have been stereo cameras on various Android models over the years, but since developers can't really count on the presence of a stereo camera for a large number of users, nothing game-changing ever really came of it, even though stereo cameras enable 3D sensing.) Though user interaction and 3D acquisition are amazingly cool, I don't think Apple would ever include a depth sensor just for those reasons. However, Apple has shown it cares pretty deeply about the iPhone as a camera and is even willing to add new hardware if it helps people take better photos, the dual LED flash being the latest example of this. So, as crazy as it sounds, I think Apple could add a back-facing depth sensor in 2-3 years mostly in order to allow for automatic depth-based post-processing of photos, mostly for better-looking lighting, all without the user having to care that there was 3D sensing involved. All the other stuff will then become possible, and widespread, as a side-effect. EDIT: On the Android front, the PrimeSense folks had a booth set up at CVPR this year and mentioned that at least one manufacturer was planning to include their Capri sensor (a miniature Kinect) in an Android tablet some time soon.
- beambot 13y ago
  I imagine depth sensing will be more useful for Glass-like devices that can really benefit from hands-free gesture recognition. It seems like a much better fit than cell phones. PS -- Hi Grant! From a fellow GT Robotics alum.
  - GrantS 13y ago
    Very true, and I'm glad that approach is being actively pursued by meta (http://www.kickstarter.com/projects/551975293/meta-the-most-advanced-augmented-reality-interface http://www.kickstarter.com/projects/551975293/meta-the-most-...). I also like the Oculus Rift hack that Occipital showed briefly here -- it makes a lot of sense for augmented/virtual reality, though I guess it means you can only interact with things when looking at your hands, as opposed to Sixense's controller-based approach. PS: Hi, Travis!
- 205guy 13y ago
  I think 2 things need to happen for depth sensing to go mainstream (and I do believe they will happen sooner or later): - Games. Look at the tennis ball example on the original article. Remember Google's IRL photo-based game (Ingress), now just imagine that with fine-grained 3D (Google will use it to crowd-source a centimeter-model of the earth). Games like this remain a bit of a niche, but just imagine if someone makes a massive social game out it on FB (a cross between Farmville, Sims, and Minecraft, projected onto your real world). Of course, someone could also create a shocking IRL FPS game (imagine your kids pointing this out the window in traffic and "shooting" at people and cars to watch them blow up). Finally something to use the processing power in theses little phones and tablets. - 3D photography. I think this is the future of photography. Take a picture of something, extract the spacial data from the image, modify it/change the p.o.v. Recall the recent image/object maniupulation video (the SIGGRAPH one that used the PatchMatch algorithm to fix the background). Each photo becomes a mini-scene that you can navigate around (kinda like the "frozen" 360 degree pans in the Matrix). Next step is time-dimension, in other words 3D immersive movies where the viewer can move around almost anywhere while the movie unfolds. You can guess the first industry to adopt this... Both of these can currently be done with flat images, processing power, and some human guidance. With depth sensing it can be faster, automated, and more accurate.
- e12e 13y ago
  > there have been stereo cameras on various Android models over the years Hm. I now envision a (physical) addon that has a two mirrors, like a periscope, and fits over the end of the phone, and the "face" camera -- allowing the two cameras to be used for stereo imaging... Not sure if you'd be able to do just as well with motion sensor data coupled with just the main camera -- and having the user move the phone from side to side (using parallax for depth sensing) ... would make for an interesting algorithmic problem at least.
rorrr2 13y ago
Nope. How often do you need to scan a 3D model of anything? I'm sure it will be popular with the creative people. It's possible it will get popular in some niche markets, like real estate agents taking 3D shots of the apartments they offer (though not much hope here, they still fail to take decent photos even now).
- GuiA 13y ago
  How often do you need to take panoramic pictures? Not that often, but it's fun and gimmicky enough that it's on the iPhone. If (when) we can make a cheap and small sensor that can provide depth data to the camera to make cool looking 3D pictures, it'll likely find its way on smartphone cameras. In 2050, will our pictures still be 2D arrays of RGB pixels?
  - rorrr2 13y ago
    Panorama thing is a software solution, it doesn't require any additional hardware.
  - mikeash 13y ago
    Perhaps more to the point, how often do you "need" to take pictures at all? Smartphone cameras have proliferated even though most people don't "need" to be able to take pictures of their food or whatever.
- noonespecial 13y ago
  How often do you need one of those auto-ma-whatsits anyway? They're really only good for a few blocks of downtown where the streets are flat cobblestone. They're useless on most trails. Just get a higher resolution horse and be happy with it.
  - rorrr2 13y ago
    > How often do you need one of those auto-ma-whatsits anyway? Every day.
- jorleif 13y ago

GrantS 13y ago

So this is a topic I have thought about for awhile -- I released a 3D scanning app for iPhone nearly 3 years ago (uses the screen as the light source for scanning, called Trimensional) and have had many conversations about when Kinect-style sensing would become standard. If we just take Apple as an example, for them to include it standard, it would have to really justify its existence. (I realize many people don't care about Apple, but I choose Apple because, for example, there have been stereo cameras on various Android models over the years, but since developers can't really count on the presence of a stereo camera for a large number of users, nothing game-changing ever really came of it, even though stereo cameras enable 3D sensing.) Though user interaction and 3D acquisition are amazingly cool, I don't think Apple would ever include a depth sensor just for those reasons. However, Apple has shown it cares pretty deeply about the iPhone as a camera and is even willing to add new hardware if it helps people take better photos, the dual LED flash being the latest example of this. So, as crazy as it sounds, I think Apple could add a back-facing depth sensor in 2-3 years mostly in order to allow for automatic depth-based post-processing of photos, mostly for better-looking lighting, all without the user having to care that there was 3D sensing involved. All the other stuff will then become possible, and widespread, as a side-effect. EDIT: On the Android front, the PrimeSense folks had a booth set up at CVPR this year and mentioned that at least one manufacturer was planning to include their Capri sensor (a miniature Kinect) in an Android tablet some time soon.

beambot 13y ago
I imagine depth sensing will be more useful for Glass-like devices that can really benefit from hands-free gesture recognition. It seems like a much better fit than cell phones. PS -- Hi Grant! From a fellow GT Robotics alum.
- GrantS 13y ago
  Very true, and I'm glad that approach is being actively pursued by meta (http://www.kickstarter.com/projects/551975293/meta-the-most-advanced-augmented-reality-interface http://www.kickstarter.com/projects/551975293/meta-the-most-...). I also like the Oculus Rift hack that Occipital showed briefly here -- it makes a lot of sense for augmented/virtual reality, though I guess it means you can only interact with things when looking at your hands, as opposed to Sixense's controller-based approach. PS: Hi, Travis!
205guy 13y ago
I think 2 things need to happen for depth sensing to go mainstream (and I do believe they will happen sooner or later): - Games. Look at the tennis ball example on the original article. Remember Google's IRL photo-based game (Ingress), now just imagine that with fine-grained 3D (Google will use it to crowd-source a centimeter-model of the earth). Games like this remain a bit of a niche, but just imagine if someone makes a massive social game out it on FB (a cross between Farmville, Sims, and Minecraft, projected onto your real world). Of course, someone could also create a shocking IRL FPS game (imagine your kids pointing this out the window in traffic and "shooting" at people and cars to watch them blow up). Finally something to use the processing power in theses little phones and tablets. - 3D photography. I think this is the future of photography. Take a picture of something, extract the spacial data from the image, modify it/change the p.o.v. Recall the recent image/object maniupulation video (the SIGGRAPH one that used the PatchMatch algorithm to fix the background). Each photo becomes a mini-scene that you can navigate around (kinda like the "frozen" 360 degree pans in the Matrix). Next step is time-dimension, in other words 3D immersive movies where the viewer can move around almost anywhere while the movie unfolds. You can guess the first industry to adopt this... Both of these can currently be done with flat images, processing power, and some human guidance. With depth sensing it can be faster, automated, and more accurate.
e12e 13y ago
> there have been stereo cameras on various Android models over the years Hm. I now envision a (physical) addon that has a two mirrors, like a periscope, and fits over the end of the phone, and the "face" camera -- allowing the two cameras to be used for stereo imaging... Not sure if you'd be able to do just as well with motion sensor data coupled with just the main camera -- and having the user move the phone from side to side (using parallax for depth sensing) ... would make for an interesting algorithmic problem at least.

rorrr2 13y ago

Nope. How often do you need to scan a 3D model of anything? I'm sure it will be popular with the creative people. It's possible it will get popular in some niche markets, like real estate agents taking 3D shots of the apartments they offer (though not much hope here, they still fail to take decent photos even now).

GuiA 13y ago
How often do you need to take panoramic pictures? Not that often, but it's fun and gimmicky enough that it's on the iPhone. If (when) we can make a cheap and small sensor that can provide depth data to the camera to make cool looking 3D pictures, it'll likely find its way on smartphone cameras. In 2050, will our pictures still be 2D arrays of RGB pixels?
- rorrr2 13y ago
  Panorama thing is a software solution, it doesn't require any additional hardware.
- mikeash 13y ago
  Perhaps more to the point, how often do you "need" to take pictures at all? Smartphone cameras have proliferated even though most people don't "need" to be able to take pictures of their food or whatever.
noonespecial 13y ago
How often do you need one of those auto-ma-whatsits anyway? They're really only good for a few blocks of downtown where the streets are flat cobblestone. They're useless on most trails. Just get a higher resolution horse and be happy with it.
- rorrr2 13y ago
  > How often do you need one of those auto-ma-whatsits anyway? Every day.
jorleif 13y ago