18 ms·
A UI Experiment with the iPhone X’s Front-Facing Camera
- peteretep 9y agoStill not available on the app store. Are there _any_ apps actually taking advantage of this yet, other than the animated emoji? Kinda bummed I've not had anything nice to play with yet
- darrenf 9y agoYes—Rainbrow, a game you control with your eyebrows: https://itunes.apple.com/us/app/rainbrow/id1312458558?mt=8 https://itunes.apple.com/us/app/rainbrow/id1312458558?mt=8
- John_KZ 9y ago>Are there _any_ apps actually taking advantage of this yet Yes, apple is harvesting your high-quality biometrics using this. 5 years down the line someone will leak the data and your face will end up on some weird porno series.
- aluhut 9y agoCritical Apple comments are risky here ;)
- macintux 9y agoStupid ones are.
- aluhut 9y agoWhat is stupid about expecting this highly personal data to leak or be misused these days...macintux...?
- macintux 9y agoThe strong implication of the original comment was that Apple was centrally collecting biometrics and would eventually sell it off or leak it. The biometrics are stored on each iPhone, in a secure enclave with severely limited access, and they're only useful for identifying (slowly) someone at close range. So, yes, theoretically in the future someone could hack all iPhones everywhere using one of the most sophisticated attacks ever, and collect data that might conceivably be useful for high resolution cameras to identify faces at close range. Maybe. Although Face ID uses infrared, so would the high resolution cameras need to have that, too? Anyway, that's why I felt it was a stupid comment.
- aluhut 9y agoThe bio-metrics shouldn't be collected in the first place. A simple patch on this closed system would allow who knows whom to collect the data. Just considering the current political situation in this companies home country it's a unreasonable risk. Considering their willingness to cooperate with dictatorships and similar ads to this. Fear not stupid at all. Considering the fact that this exists because a) Apple was unable to come up with any real innovation to sell the new product and b) people seem to be really to lazy to type a number and rather give up that information. Data Minimization, Data Avoidance are a thing. > "using one of the most sophisticated attacks ever" As if we haven't heard that speech before...
- macintux 9y agoUsers worried about their privacy can always choose not to use it. The more likely attack vector is simply capturing scenes through the front camera. That would give you, most of the time, an image of the face of the user, not just depth mapping information of questionable value. But really, nearly everyone shares images of themselves online, so even that's of dubious value. I'd challenge you to find any security expert who agrees that Face ID as Apple implements it can realistically result in useful biometric data being leaked to a hostile party. Apple supplies whitepapers documenting the secure enclave, I imagine there's one for Face ID. Now, the question of whether Face ID is secure enough for any given user's needs as a local authentication is a perfectly valid question, and clearly for some users the answer is no. But, again, it's optional, and that's not at all the threat under discussion.
- tinus_hn 9y agoParanoia aside the data does not leave the device. All that is available to apps are the 2d images seen by the cameras. These are hardly special.
- yeldarb 9y agoYes, we released Nose Zone: https://nose.zone https://nose.zone You’ll probably see more coming soon. Ours was in review by Apple for over a month.
- eric_h 9y ago> Ours was in review by Apple for over a month. ?! That's crazy talk - I've not seen a review process for a new app take more than a few days in recent memory. I guess using the Face ID API marked your app for more thorough review?
- yeldarb 9y agoYeah, they are pretty thoroughly vetting any app that uses the TrueDepth APIs. Even had to go as far as talking with them on the phone a couple times to reassure them we weren't using "face data" in any nefarious ways. (We're not; everything is kept on device)
- eric_h 9y agoThat sucks - I was on a team that released a fully legal (in the USA) gambling app a few years back and while we had to have a few back and forths, it definitely didn't take a month to get approved. At least you finally got it out!
- eric_h 9y agoIt just occurred to me that I'm not sure how they can even be sure that you were telling them the truth. Not that I'm suggesting otherwise, but within the scope of the tools they have available to them while doing the review, what's to prevent you from saying "sure, we're keeping everything on device" when in fact you're building an international database of faces that you intend to sell to a nefarious government actor.
- yeldarb 9y agoYeah very true. One of the things they did was make us explicitly state in our privacy policy how (and what for) we used data from the TrueDepth APIs. So I guess at least someone would have legal recourse against us in that case.
- gambiting 9y ago>>The notch. Everyone hates it. These broad statements irritate me. I don't hate it - I actually think it integrates super well with the UI and makes the X look unique among the sea of identically looking glass slabs. And I don't even own an iphone.
- mrmondo 9y agoAs it does me, who are they to say what I hate? I most certainly don’t hate the ‘notch’ - I couldn’t care less about it other than hey - I’ve got some extra screen real estate on either side of the speaker and cameras so that the clock and indicators don’t have to cut into application space.
- monk_e_boy 9y agoTech journalists who are always looking for the next best thing and just HAVE to have a strong opinion on everything seem very reluctant to embrace change. No one I know cares very much, they think it's neat, they all wonder at the manufacturing of the oled screen, but like you, they like the extra screen real estate. I feel there must be some sort of 'too cool for skool' bubble that journalists live in.
- baddox 9y agoI certainly don’t hate it either, but I also wouldn’t say I like it specifically. I love the phone altogether, and the notch doesn’t ever bother me, but sure, in theory it would be better if all the same features could be implemented without a notch. I recognize it as a reasonable design compromise that is justified and necessary given current technology.
- amrrs 9y agoAnd it looks like it's creating a cult - https://www.gsmarena.com/the_notch_is_spreading_for_no_good_reason_the_mwc_was_rife_with_iphone_x_lookalikes-news-29909.php https://www.gsmarena.com/the_notch_is_spreading_for_no_good_...
- tinus_hn 9y agoEverybody hates it so much the other manufacturers are scrambling to copy it. https://www.theverge.com/2018/3/4/17077458/iphone-design-clones-mwc-2018 https://www.theverge.com/2018/3/4/17077458/iphone-design-clo...
- mabedan 9y agoI'm pretty sure the identical effect can be (and has been) produced by the combination of normal front facing camera, accelerometer and gyroscope.
- gabrielgoh 9y agoI don't think so. This effect, at least as described, can change even as the phone remains static, e.g. if you head moves while the phone sits still on the table.
- ponyous 9y agoParent's comment mentions front facing camera. Probably for the same reason.
- baddox 9y agoThe only required piece of information to render the scene is the position of the eye relative to the phone, which can be represented as an x and y coordinate relative to the rectangular frame captured by the front-facing camera. The accelerometer and gyroscope shouldn’t be needed, except perhaps to smooth out movement of the phone in the user’s hand if the camera’s frame rate is too low. My guess is that the only reason this demo is relying on the iPhone X is that Apple provides a facetracking API, so there’s no need to write any code or use external libraries to do eyetracking with a normal front facing camera.
- rocqua 9y agoTo really get a nice 3D effect you also need distance of the eyes from the phone. Based on the video in the article, this demo actually uses this depth information. You can't get this depth info easily from a camera. I can certainly see how the more advanced optics in the iPhone X really help here.
- baddox 9y agoI suspect the size of the phone screen isn’t large enough for distance information to affect the rendered output significantly.
- d--b 9y agoThis is neat. I think what would be even neater is to use the head tracking from the notch to compute a transformation of the back-side camera to simulate that you can see through the iPhone. Of course this wouldn't work when the iPhone is too close, or if the angle is too wide, but in other cases, this would make AR really good.
- 220V_USKettle 9y agoI assume with two cameras on the back, it would be possible to create a stereo video stream and implement a transformation as you described.
- neelkadia 9y agoThis is very similar to Johnny Lee's initial exploration with Wii... http://johnnylee.net/projects/wii/ http://johnnylee.net/projects/wii/
- ChrisRR 9y agoThis reminds of the head tracking demo using a Wii remote made by Johnny Lee years ago. https://www.youtube.com/watch?v=Jd3-eiid-Uw https://www.youtube.com/watch?v=Jd3-eiid-Uw
- ladberg 9y agoThis is mentioned in the article too...
- mft_ 9y agoLOL, remember getting that working at home, back in the day - gyrating in front of my monitor with (from memory) the Wii's IR transmitter bar held to my forehead. It was a lovely little taste of what might have been.. although we're still waiting!
- kakarot 9y agoI was more fascinated with his projected touchscreen using a pressure-activated LED at the end of a pen. He's quite the visionary.
- Lewton 9y agoI'm torn, there's literally a link to Johnny Lee's work in the article. So your comment doesn't add anything. But a lot of people read only the comments (or the comments before the article) so your comment does add value Is it a good HN comment or not? :D
- bringtheaction 9y agoIt’s an acceptable kind of HN comment. A link in the article is generally easy to miss and probably others missed it too. Admittedly the article did talk enough about the other thing that it’d be pretty hard to miss if you read the whole thing, but I think we can forgive it still. Perhaps the parent to your comment opened the page and saw the video but didn’t bother to read the article. I think that’s fine too. I always read comments first and in fact quite often I read the comments only. As long as we don’t derail the discussion completely by starting a top-level comment about how we perceive something based on the title alone I think it’s fine. That includes it being fine to mention things that you are reminded of by the title as long as you phrase it correctly. In my opinion of course.
- Shivetya 9y agoVery cool but I do have to laugh at his opening remark, the notch and how everyone hates it. If anything screams that Android phone makers have no imagination it is the notch, suddenly its on their phones. I do think the notch is bad UI but it can be worked around. I would have more easily accepted a narrower band of black across the top and bottom but Apple isn't losing sales because the rest of the phone's tech is that good. I do look forward to the day when the components are barely distinguishable from the display.
- annabellish 9y ago_Some_ android phone manufacturers are adding pointless notches, but it's a big market. You'll find some android manufacturer doing basically anything. Personally I find Samsung's current designs prettier than the iPhone X, though not by much - thing's gorgeous.
- dingo_bat 9y ago> but Apple isn't losing sales because the rest of the phone's tech is that good. Correction: Apple isn't losing sales because people will literally buy a bag of shit for $1000 if it is branded apple. Name me 5 areas where the tech in iPhone is superior to Samsung's flagship.
- annabellish 9y agoI mean, 1. Face Id is wildly superior to anything else on the market 2. Apple's processors are still better than either Snapdragon or Exynos chips, though the latter is closing in 3. They still get a lot further on less RAM and battery 4. The iPhone X screen somewhat beats the S8/N8, though potentially not the S9 5. Their force touch implementation is better than Samsung's, too. I love my note 8 and wouldn't part with it for any iOS device, but suggesting they're entirely noncompetitive is silly.
- inventingwithmo 9y agoThis looks amazing when you view it via a video, but the effect IRL is nowhere near as compelling. The lack of parallax smokes it.
- deleted 9y ago[deleted]
- russellbeattie 9y agoThe much maligned Amazon Fire Phone had four specialized high-speed head tracking cameras on the front of it, and made this effect the centerpiece of it's UI. I worked on it, and it was very cool. The phone failed for lots of reasons, but this particular "gimmick" actually held some promise. The example clay-molding game which came with it actually showed how being able to perceive the image in pseudo-3D did have a real application.
- berdon 9y agoI worked on it too. You're right about the head tracking - it did work very well. They had some great UX ideas centered around it too. It's unfortunate it didn't pan out for those UX features alone.
- subculture 9y agoWhich were the great UX ideas? I think I missed them... The fact that you had to move your head or phone just to see the time was bad. The 'peek' gesture which required moving your head to the side and looking sidelong at the phone to open a side navigation menu was equally lame.
- russellbeattie 9y agoTo be self critical, I think there was a lot of, "We've got this cool hammer, let's make everything a nail" sort of thing going on where the perspective stuff was grafted onto a lot of areas better left alone. That said, the tilt scrolling on web pages was actually very nice to use (remember it was based on the angle to your eyes, not the physical phone angle, so you could use it lying down). And like I said, applications where it was helpful to glance around a 3D object, was quite compelling. But yeah, there were too many areas where the effect was used where it probably didn't need to be.
- berdon 9y agoThe feature I miss/wish for the most was looking at the phone sitting on your desk and having the screen turn on. "Wonder if I have any notifications...Nope"
- drosan 9y agoIs it a joke? "Wild experiment"? "first to take the effect mainstream"? What kind of bullshit is that? The said effect and the head tracking principle was already used in the New Nintendo 3DS consoles that was sold millions worldwide. Not only that - 3DS' top screen has a true "parallax barrier" screen, means it displays a 3D picture with depth and all.
- sorrybut 9y agoApple are the first to innovate all tech when one's entire worldview is limited to the Apple product line.
- digi_owl 9y agoAnd because Apple has the media/design world by the balls thanks to ingrained habits, they have blinders on for the rest of the tech world.
- deleted 9y ago[deleted]
- DiThi 9y agoN3DS face tracking is only used to correct the stereo separation (so each eye sees a separate image without having to be exactly on the center), not to move the scene depending on where you are. So if you have a scene like in that experiment, you would see it skew to the opposite direction you move your head. Also the vast majority of games don't have that much depth anyway, even at the highest setting, probably because the scene doesn't move accordingly and would make people dizzy.
- drosan 9y agoYet it was essentially the same technology that was named "wild experiment" in the article above. Also visually the scene does move accordingly, mostly because the top display is tecnhically two displays in one and they render the scene from 2 different angles.
- sundvor 9y agoAs much as I don't like the iPhone Notch and all that, the video in the article was quite cool, appeared very performant and a great programming showcase. So kudos to the Dev for making this.
- cjmoran 9y agoI've always found these pseudo-3D effects (including the slight "floating-icons" effect applied to iOS wallpapers) to be unconvincing in real life. I suspect they look much better in demo videos like this because there's no depth information in the objects surrounding the phone either -- the person's hand, the floor/desk behind them, etc. This makes it much easier for one's brain to map "pseudo-depth" onto the entire video. In real life, you have depth perception for everything except the images on your phone's screen, making a demo like this look "flat" even at its most convincing. Real 3D requires two distinct visual inputs; this is a gimmick.
- adamscybot 9y ago"The notch. Everyone hates it. And what did we get for it? A rooster that sings karaoke. No thanks." This isn't true. What we got for it is Face ID and the consequential removal of the home button which then allowed for the frameless design. Not saying it's ideal. But the author has missed the point slightly.
- joshstrange 9y agoYeah, everyone hates it so much that a ton of android phones being released this year have their own notch... It really does just fade away for me, I don't notice it anymore
- TremendousJudge 9y agoand the essential phone, that had a notch before it was cool
- duncans 9y agoI see it not as a notch but some bonus bits of screen for the clock and signal/battery status.
- hammock 9y agoSamsung S8 has face ID and no home button...
- adamscybot 9y agoThat's just facial recognition via an image, not a 3d IR scanner. As well as that, it effectively has a "full width" notch at the top.
- 220V_USKettle 9y agoHow difficult is it to use/implement as a 3D scanner (3d modeling)?
- soylentcola 9y agoHaven't messed with the depth sensor on an iPhone yet, but you could do it with Kinects and other depth cams like the Intel Realsense series. In my experiments with those, the main drawback was ergonomics (device of varying size depending on the sensor, connected to a computer via at least one cable as you moved it around the thing you were scanning). On a phone or other standalone/self-contained bit of hardware, I'd imagine it makes things easier.
- mcphage 9y ago> Haven't messed with the depth sensor on an iPhone yet, but you could do it with Kinects and other depth cams Well, if you’ve used a Kinect, then you’ve used the iPhone’s tech, because Apple bought PrimeSense, which developed the initial Kinect for Microsoft.
- jordache 9y agoYo mark willson of co.design. Were you around 10 years ago? Let's tame the article title a bit huh? https://www.youtube.com/watch?v=Jd3-eiid-Uw https://www.youtube.com/watch?v=Jd3-eiid-Uw
- giarc 9y agoHe talked about Johnny in the article. >Norrby is far from the first person to play with trompe-l’oeil (also called “parallax”) in the modern age. Johnny Lee, now at Google working on augmented reality, first made his mark by turning a stock television and a Nintendo Wiimote into a wild 3D game.
- jordache 9y agooh ok.. so he should know it's not 'wild'. annoying bait title.
- oceanswave 9y agoWhat’s the difference between this and the plethora of AR games and apps that ‘trick your mind to believing there’s a whole world beyond the screen’ Using face positioning vs the accelerometer is interesting, but without seeing it in action, is it that much more compelling?