4 ms·

He talks about how "Gaze & pinch is an incredible UI superpower and major industry ah-ha moment" but... if that's really the case, then it's quite an indictment

He talks about how "Gaze & pinch is an incredible UI superpower and major industry ah-ha moment" but... if that's really the case, then it's quite an indictment of the VR industry: > The hardware needed to track eyes and hands in VR has been around for over a decade, and it’s Apple unique ability to bring everything together in a magical way that makes this UI superpower the most important achievement of the entire Vision Pro product, without a shadow of doubt. So they had all the pieces, but only Apple put it together and realized that you'd need a VR equivalent of point-and-click? If that's actually true, it's sad.

crooked-v 3y ago
It's almost exactly the same kind of conceptual transition that Apple made happen with keyboardless smartphones, too, which adds an extra sort of funny element to it.
sumedh 3y ago
> So they had all the pieces, but only Apple put it together Its very difficult to change a mindset or culture in big companies. Existing VR companies were too invested in using a controller. Similarly back in the early smartphone days all the big companies thought that smartphones must have physical keyboard.
- tpmoney 3y ago
  Sometimes you really have to decide to take the risk and ship without a standard controller before you can see if the new model will work. The iPhone was famously derided for a lack of stylus. Video game consoles for years have tried to incorporate motion controls in some form or another and realistically only the Wii succeeded in any measure because they ditched the classic controller instead of trying to shoehorn it in. Many times it doesn't work, but if you give developers and users an "easy escape hatch" to go back to what they're already comfortable with, so many of them will default to that no matter how much better your new option might be.
MBCook 3y ago
On top of what the others said, there are two other closely related issues: If everything is designed for your controller, the eye interface may not work well due to lack of software optimization. Which means it’s just an expensive battery hogging extra weight you don’t need.
jimbokun 3y ago
Maybe they realized they needed it, but Apple actually pulled it off. Apple has a stronger combination of hardware design, software implementation skills, and UX expertise, than any company in the world.
interpol_p 3y ago
Putting it together is not as simple as it seems. I think it was an immense engineering and design effort from Apple to get it to the point where it feels effortless and obvious Not only do they have two cameras per eye, and all the hardware for wide angle out-of-view hand tracking, they had to consider: Privacy: the user’s gaze is never delivered to your process when your native UI reacts to their gaze. Building this infrastructure to be performant, bug free and secure is a lot of work. Not to mention making it completely transparent for developers to use Design: they reconsidered every single iOS control in the context of gaze and pinch, and invented whole new UI paradigms that work really well with the existing SDK. You can insert 3D models into a SwiftUI scroll view, and scroll them, and it just works (they even fade at the cut off point) Accessibility: there is a great deal of thought put into alternative navigation methods for users who cannot maintain consistent gaze In addition to this they clearly thought about how to maintain “gazeable” targets in the UI. When you drag a window closer or farther it scales up and down maintaining exactly the same visual size, trying to ensure nothing gets too small or large to gaze at effectively There are so many thousands of design and engineering decisions that went into making gaze and pinch based navigation work so simply, so I can understand how it hasn’t been done this effectively until now

crooked-v 3y ago

It's almost exactly the same kind of conceptual transition that Apple made happen with keyboardless smartphones, too, which adds an extra sort of funny element to it.

sumedh 3y ago

> So they had all the pieces, but only Apple put it together Its very difficult to change a mindset or culture in big companies. Existing VR companies were too invested in using a controller. Similarly back in the early smartphone days all the big companies thought that smartphones must have physical keyboard.

tpmoney 3y ago
Sometimes you really have to decide to take the risk and ship without a standard controller before you can see if the new model will work. The iPhone was famously derided for a lack of stylus. Video game consoles for years have tried to incorporate motion controls in some form or another and realistically only the Wii succeeded in any measure because they ditched the classic controller instead of trying to shoehorn it in. Many times it doesn't work, but if you give developers and users an "easy escape hatch" to go back to what they're already comfortable with, so many of them will default to that no matter how much better your new option might be.

MBCook 3y ago

On top of what the others said, there are two other closely related issues: If everything is designed for your controller, the eye interface may not work well due to lack of software optimization. Which means it’s just an expensive battery hogging extra weight you don’t need.

jimbokun 3y ago

Maybe they realized they needed it, but Apple actually pulled it off. Apple has a stronger combination of hardware design, software implementation skills, and UX expertise, than any company in the world.

interpol_p 3y ago

Putting it together is not as simple as it seems. I think it was an immense engineering and design effort from Apple to get it to the point where it feels effortless and obvious Not only do they have two cameras per eye, and all the hardware for wide angle out-of-view hand tracking, they had to consider: Privacy: the user’s gaze is never delivered to your process when your native UI reacts to their gaze. Building this infrastructure to be performant, bug free and secure is a lot of work. Not to mention making it completely transparent for developers to use Design: they reconsidered every single iOS control in the context of gaze and pinch, and invented whole new UI paradigms that work really well with the existing SDK. You can insert 3D models into a SwiftUI scroll view, and scroll them, and it just works (they even fade at the cut off point) Accessibility: there is a great deal of thought put into alternative navigation methods for users who cannot maintain consistent gaze In addition to this they clearly thought about how to maintain “gazeable” targets in the UI. When you drag a window closer or farther it scales up and down maintaining exactly the same visual size, trying to ensure nothing gets too small or large to gaze at effectively There are so many thousands of design and engineering decisions that went into making gaze and pinch based navigation work so simply, so I can understand how it hasn’t been done this effectively until now