4 ms·

(Caveat: I don't know a lot about pass-through HMD AR, and I assume you're incredibly smart and that the patent is innovative.) > …it's a little bit refreshing

(Caveat: I don't know a lot about pass-through HMD AR, and I assume you're incredibly smart and that the patent is innovative.) > …it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology… I understand the framing as "compensatory technology", but is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. But a 100% digital vision pipeline seems like it could unlock interesting capabilities like "night vision", new ways of highlighting interesting objects, etc.

gjsman-1000 3y ago
I think the most prominent attempt at see-through AR was the Microsoft HoloLens. But if you've actually tried the HoloLens, the Field of View is atrociously, tragically small. The first HoloLens had a field of view of 30°*17.5°. The second HoloLens improved to 43°*29°, but it's still best described as "cramped." Couple that with almost all of the compute budget for the device going into vision processing and having very little compute left for actually running apps (the first HoloLens having a 1Ghz Intel Atom from 2015, the second a superior... Snapdragon 850). The other problem, of course, is that nothing can be truly solidly-colored. Everything has some opacity - which, combined with the FOV issue, is why HoloLens was never marketed as having anything to do with VR.
- moron4hire 3y ago
  The vision processing was done with a separate SoC. Your applications were not contending for compute time with the vision processing. That's just how anemic the CPU was.
- dfsl 3y ago
  [dead]
AndrewKemendo 3y ago
In theory the wavefront emulator is the simplest display - all of the "pixels" are rendered in your brain so you don't have to build an actual "display" HOWEVER if someone can get the input -> photon production pipeline to be less than ~10ms, then that does solve a lot of the rendering issues, however it doesn't solve all of the other long term problems that come with that amount of hardware - including weight and complexity. That said, there's a lot of known-unknowns that need to be solved, for example I don't have a solution for micropiezo resonance issues that I'm sure will crop up.
alfalfasprout 3y ago
Agreed. In fact, this approach is also likely better for military applications which I hope they explore. Currently, NVGs that are fielded by soldiers are already displaying images in a way that's not pass-through (using classic image intensification tubes). Something like the Apple vision headset in a lighter and more durable form factor would allow for eg; fusion imagery (fusing visible, thermal, and night vision).
- Despegar 3y ago
  It's not in Apple's interests to get into military contracting because of the potential to damage their brand, as well as avoiding the geopolitical tensions that would invite.
  - alfalfasprout 3y ago
    Apple won't. But as soon as these headsets hit the market manufacturers will be tearing them apart and reverse engineering them. $4k/unit is nothing when it comes to military hardware. There's also no need for the outward facing display, etc. If anything, I wouldn't be surprised if Hololens goes in this direction. And the DoD already has contracts around the hololens.
- numpad0 3y ago
  I’ve always wanted an NVG to play with, and from watching firearms nerd rants I’ve learned: - real NVG costs way above my budget, and all comes with defects. Realistically the tubes in civilian markets are all defective returns. - even real NVGs can’t work in complete darkness, such as in a closet; moonlight or starlight is needed. - real NVGs on firearms are often used in conjunction with an IR laser and a flashlight for aiming; advantage is solely that you have to have a goggle to see it, not that you can operate without making any emissions. - partly due to above points, plain old CMOS cameras with modified 1” Sony sensors for better low light performance is starting to match practical usefulness of a real NVG(it’s Cold War tech so no wonder here). Latency is an issue, but more of “is an issue” and no longer a blocker. … Vision Pro type of devices replacing NVGs indeed seem like a near future possibility.
jolmg 3y ago
> is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. I think they're different and not one better than the other. If I wanted to drive with an HMD on, or otherwise be in a situation where it could be deadly to have my sight turned off for even a second or to even have some lag or stutter or other glitch in my eyesight, I'd much rather have a pass-through AR HMD. One's sense of sight seems much more reliable with it by its very nature. You simply don't have those modes of failure with transparent plastic, no matter what's going on in the hardware/software.
- flangola7 3y ago
  Your steering wheel and pedals are digital too. Airliner controls have been digital for decades. Factories have life or death safety systems that count on silicon and a software stack to respond quickly and without error. Apple can make it possible if they wanted too.
  - faitswulff 3y ago
    These are all very different from the “HUD goes dark and my meat brain panics, causing me to get into an accident” scenario that GP is talking about.
    - YetAnotherNick 3y ago
      There are multiple system running on silicon that have been made to not go dark in normal mode of operation. Some examples could be found at https://en.wikipedia.org/wiki/Safety-critical_system https://en.wikipedia.org/wiki/Safety-critical_system
  - deleted 3y ago
    [deleted]
  - sdeframond 3y ago
    I believe cars' steering are analog because it would too expensive to achieve the required level of reliability. Planes are crazy expensive anyway,so fly-by-wire can be afforded.

gjsman-1000 3y ago

I think the most prominent attempt at see-through AR was the Microsoft HoloLens. But if you've actually tried the HoloLens, the Field of View is atrociously, tragically small. The first HoloLens had a field of view of 30°*17.5°. The second HoloLens improved to 43°*29°, but it's still best described as "cramped." Couple that with almost all of the compute budget for the device going into vision processing and having very little compute left for actually running apps (the first HoloLens having a 1Ghz Intel Atom from 2015, the second a superior... Snapdragon 850). The other problem, of course, is that nothing can be truly solidly-colored. Everything has some opacity - which, combined with the FOV issue, is why HoloLens was never marketed as having anything to do with VR.

moron4hire 3y ago
The vision processing was done with a separate SoC. Your applications were not contending for compute time with the vision processing. That's just how anemic the CPU was.
dfsl 3y ago
[dead]

AndrewKemendo 3y ago

In theory the wavefront emulator is the simplest display - all of the "pixels" are rendered in your brain so you don't have to build an actual "display" HOWEVER if someone can get the input -> photon production pipeline to be less than ~10ms, then that does solve a lot of the rendering issues, however it doesn't solve all of the other long term problems that come with that amount of hardware - including weight and complexity. That said, there's a lot of known-unknowns that need to be solved, for example I don't have a solution for micropiezo resonance issues that I'm sure will crop up.

alfalfasprout 3y ago

Agreed. In fact, this approach is also likely better for military applications which I hope they explore. Currently, NVGs that are fielded by soldiers are already displaying images in a way that's not pass-through (using classic image intensification tubes). Something like the Apple vision headset in a lighter and more durable form factor would allow for eg; fusion imagery (fusing visible, thermal, and night vision).

Despegar 3y ago
It's not in Apple's interests to get into military contracting because of the potential to damage their brand, as well as avoiding the geopolitical tensions that would invite.
- alfalfasprout 3y ago
  Apple won't. But as soon as these headsets hit the market manufacturers will be tearing them apart and reverse engineering them. $4k/unit is nothing when it comes to military hardware. There's also no need for the outward facing display, etc. If anything, I wouldn't be surprised if Hololens goes in this direction. And the DoD already has contracts around the hololens.
numpad0 3y ago
I’ve always wanted an NVG to play with, and from watching firearms nerd rants I’ve learned: - real NVG costs way above my budget, and all comes with defects. Realistically the tubes in civilian markets are all defective returns. - even real NVGs can’t work in complete darkness, such as in a closet; moonlight or starlight is needed. - real NVGs on firearms are often used in conjunction with an IR laser and a flashlight for aiming; advantage is solely that you have to have a goggle to see it, not that you can operate without making any emissions. - partly due to above points, plain old CMOS cameras with modified 1” Sony sensors for better low light performance is starting to match practical usefulness of a real NVG(it’s Cold War tech so no wonder here). Latency is an issue, but more of “is an issue” and no longer a blocker. … Vision Pro type of devices replacing NVGs indeed seem like a near future possibility.

jolmg 3y ago

> is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. I think they're different and not one better than the other. If I wanted to drive with an HMD on, or otherwise be in a situation where it could be deadly to have my sight turned off for even a second or to even have some lag or stutter or other glitch in my eyesight, I'd much rather have a pass-through AR HMD. One's sense of sight seems much more reliable with it by its very nature. You simply don't have those modes of failure with transparent plastic, no matter what's going on in the hardware/software.

flangola7 3y ago
Your steering wheel and pedals are digital too. Airliner controls have been digital for decades. Factories have life or death safety systems that count on silicon and a software stack to respond quickly and without error. Apple can make it possible if they wanted too.
- faitswulff 3y ago
  These are all very different from the “HUD goes dark and my meat brain panics, causing me to get into an accident” scenario that GP is talking about.
  - YetAnotherNick 3y ago
    There are multiple system running on silicon that have been made to not go dark in normal mode of operation. Some examples could be found at https://en.wikipedia.org/wiki/Safety-critical_system https://en.wikipedia.org/wiki/Safety-critical_system
- deleted 3y ago
  [deleted]
- sdeframond 3y ago
  I believe cars' steering are analog because it would too expensive to achieve the required level of reliability. Planes are crazy expensive anyway,so fly-by-wire can be afforded.