6 ms·

Having patented technology for see through AR display in 2016 that is cited by Apple [1] and knowing how crazy hard it is, it's a little bit refreshing to know

Having patented technology for see through AR display in 2016 that is cited by Apple [1] and knowing how crazy hard it is, it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology instead of trying to solve the hard see through AR problems. [1]https://patents.google.com/patent/US10757400B2/en https://patents.google.com/patent/US10757400B2/en

warning26 3y ago
I'm quite certain they're still trying to solve the hard see-through AR problems, and are hoping to release a future Vision headset with a true see-through display. But otherwise I agree, it makes sense for them to focus on what they can do best with current tech as a stopgap. With current see-through HMD tech, AR ends up incredibly disappointing. (See also: Hololens & Magic Leap's limited FOV)
- AndrewKemendo 3y ago
  Right, I think it's going to come eventually but it's really really really hard
- samwillis 3y ago
  My understanding is that occlusion in "see through" AR is an unsolved problem, everything looks ghostly and somewhat semitransparent. Until someone solves that I suspect the re-projection method is the only viable option.
  - AndrewKemendo 3y ago
    Yes "projecting black" is effectively impossible - every once in a while someone from Magic Leap would claim they had solved it but everyone knew it was bullshit
    - jessriedel 3y ago
      I'm sorta surprised by this. Don't liquid crystal displays in laptops do exactly that?: LCDs are a controllably transparent array that is overlaid on a uniform backlight. So why can't one create a set of glasses where the lens are an LCD array without the backlight?
      - AndrewKemendo 3y ago
        An LCD simply turns "off" to display "black" - so the emissions stop (relative to surrounding pixel emissions) With a projector, you can't "throw" nothing (aka black). As a result "projected black" is simply lack of projection. In the case of a translucent or transparent reflection or waveguide surface - which is what the projection reflects off of - "black" is whatever the darkest part of the surface is. In effect whatever else is emitting from the surface that you're looking at will change the depth of "black" you get. This is why the Hololens and other see through AR devices are always tinted, to set a higher threshold for "black" than the surrounding unaided view.
        moron4hire 3y ago
        LCDs do not emit light. They have a light emitter (or reflector, for passive monochrome displays ala an LCD watch face) behind them and the liquid crystal part selectively allows that light to be blocked or pass through. The are three layers of polarizing material. The two outer layers are at right-angle polarizations to each other and normally would be completely opaque on their own. When power is applied to the liquid crystal, it twists the crystal's polarization to be at a 45° angle to the other two layers, which then permits some of the incident light to pass through. An optically transparent waveguide display can use an LCD layer to block light coming through the front and then not render graphics on that area of the display. It will be opaque black at that point (though rather fuzzy around the edges, as the LCD won't be in focus). Magic Leap 2 actually employs this technique. It's... a lot like the rest of the device: a good idea on paper.
        AndrewKemendo 3y ago
  - daniel_reetz 3y ago
    I work in the space. Occlusion only matters if it matters. "Ghostly" information display is suitable for a shockingly wide range of tasks.
    - samwillis 3y ago
      While that's true, I believe these (mass market) devices need to be general purpose, and so need to cover the use cases where occlusion is necessary.
    - K0balt 3y ago
      This is also what I intuited. Can you give some examples of things that require “projected black” , especially those that couldn’t be solved by using a darkened room?
      - deleted 3y ago
        [deleted]
- mensetmanusman 3y ago
  They are still asking suppliers to solve the technology, yes.
- birdyrooster 3y ago
  Hopefully we are not forgetting about Bosch's Retinal Projection which solves many problems with lensing HMD AR. If I was a betting man I would say Bosch came up with this tech almost specifically for Apple to integrate it into interactive glasses.
- dfsl 3y ago
  [dead]
nickelbob 3y ago
Just curious - why is it so hard?
- slg 3y ago
  I'm not aware of all the details around the technical complications, but from a physics perspective, you can't make something darker and more opaque by adding more light. Therefore, an AR headset needs to be able to at least partially block light in order to make convincing images. It seems a lot easier just to go with Apple's approach of blocking all light rather than try to develop tech that will only selectively block the light behind the AR objects while allowing other light through. The blocking all light approach also allows you to hide other potential weaknesses of a device. For example, a lower field of view is much more distracting in a pass-through AR device as you still have your full peripheral vision. VR devices will generally black out the light outside of the FOV making it easier to ignore.
  - ddalex 3y ago
    > you can't make something darker and more opaque by adding more light I mean... theoretically you can if you can match photon for photon with the inverse phase. Difficult, I know :)
- AndrewKemendo 3y ago
  Off the top of my head: 1. Field of view is limited to existing optics miniaturization 2. Subtractive shading (rendering black) might not be solvable 3. Variable focus objects in the same scene requires projecting n>2 significantly different wavefronts - not solved how to do this with a single vibrating element
  - pornel 3y ago
    Do you know why an LCD panel can't be used to block light?
    - Tarragon 3y ago
CharlesW 3y ago
(Caveat: I don't know a lot about pass-through HMD AR, and I assume you're incredibly smart and that the patent is innovative.) > …it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology… I understand the framing as "compensatory technology", but is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. But a 100% digital vision pipeline seems like it could unlock interesting capabilities like "night vision", new ways of highlighting interesting objects, etc.
- gjsman-1000 3y ago
  I think the most prominent attempt at see-through AR was the Microsoft HoloLens. But if you've actually tried the HoloLens, the Field of View is atrociously, tragically small. The first HoloLens had a field of view of 30°*17.5°. The second HoloLens improved to 43°*29°, but it's still best described as "cramped." Couple that with almost all of the compute budget for the device going into vision processing and having very little compute left for actually running apps (the first HoloLens having a 1Ghz Intel Atom from 2015, the second a superior... Snapdragon 850). The other problem, of course, is that nothing can be truly solidly-colored. Everything has some opacity - which, combined with the FOV issue, is why HoloLens was never marketed as having anything to do with VR.
  - moron4hire 3y ago
    The vision processing was done with a separate SoC. Your applications were not contending for compute time with the vision processing. That's just how anemic the CPU was.
  - dfsl 3y ago
    [dead]
- AndrewKemendo 3y ago
  In theory the wavefront emulator is the simplest display - all of the "pixels" are rendered in your brain so you don't have to build an actual "display" HOWEVER if someone can get the input -> photon production pipeline to be less than ~10ms, then that does solve a lot of the rendering issues, however it doesn't solve all of the other long term problems that come with that amount of hardware - including weight and complexity. That said, there's a lot of known-unknowns that need to be solved, for example I don't have a solution for micropiezo resonance issues that I'm sure will crop up.

warning26 3y ago

I'm quite certain they're still trying to solve the hard see-through AR problems, and are hoping to release a future Vision headset with a true see-through display. But otherwise I agree, it makes sense for them to focus on what they can do best with current tech as a stopgap. With current see-through HMD tech, AR ends up incredibly disappointing. (See also: Hololens & Magic Leap's limited FOV)

AndrewKemendo 3y ago
Right, I think it's going to come eventually but it's really really really hard
samwillis 3y ago
My understanding is that occlusion in "see through" AR is an unsolved problem, everything looks ghostly and somewhat semitransparent. Until someone solves that I suspect the re-projection method is the only viable option.
- AndrewKemendo 3y ago
  Yes "projecting black" is effectively impossible - every once in a while someone from Magic Leap would claim they had solved it but everyone knew it was bullshit
  - jessriedel 3y ago
    I'm sorta surprised by this. Don't liquid crystal displays in laptops do exactly that?: LCDs are a controllably transparent array that is overlaid on a uniform backlight. So why can't one create a set of glasses where the lens are an LCD array without the backlight?
    - AndrewKemendo 3y ago
      An LCD simply turns "off" to display "black" - so the emissions stop (relative to surrounding pixel emissions) With a projector, you can't "throw" nothing (aka black). As a result "projected black" is simply lack of projection. In the case of a translucent or transparent reflection or waveguide surface - which is what the projection reflects off of - "black" is whatever the darkest part of the surface is. In effect whatever else is emitting from the surface that you're looking at will change the depth of "black" you get. This is why the Hololens and other see through AR devices are always tinted, to set a higher threshold for "black" than the surrounding unaided view.
      - moron4hire 3y ago
        LCDs do not emit light. They have a light emitter (or reflector, for passive monochrome displays ala an LCD watch face) behind them and the liquid crystal part selectively allows that light to be blocked or pass through. The are three layers of polarizing material. The two outer layers are at right-angle polarizations to each other and normally would be completely opaque on their own. When power is applied to the liquid crystal, it twists the crystal's polarization to be at a 45° angle to the other two layers, which then permits some of the incident light to pass through. An optically transparent waveguide display can use an LCD layer to block light coming through the front and then not render graphics on that area of the display. It will be opaque black at that point (though rather fuzzy around the edges, as the LCD won't be in focus). Magic Leap 2 actually employs this technique. It's... a lot like the rest of the device: a good idea on paper.
        AndrewKemendo 3y ago
- daniel_reetz 3y ago
  I work in the space. Occlusion only matters if it matters. "Ghostly" information display is suitable for a shockingly wide range of tasks.
  - samwillis 3y ago
    While that's true, I believe these (mass market) devices need to be general purpose, and so need to cover the use cases where occlusion is necessary.
  - K0balt 3y ago
    This is also what I intuited. Can you give some examples of things that require “projected black” , especially those that couldn’t be solved by using a darkened room?
    - deleted 3y ago
      [deleted]
mensetmanusman 3y ago
They are still asking suppliers to solve the technology, yes.
birdyrooster 3y ago
Hopefully we are not forgetting about Bosch's Retinal Projection which solves many problems with lensing HMD AR. If I was a betting man I would say Bosch came up with this tech almost specifically for Apple to integrate it into interactive glasses.
dfsl 3y ago
[dead]

nickelbob 3y ago

Just curious - why is it so hard?

slg 3y ago
I'm not aware of all the details around the technical complications, but from a physics perspective, you can't make something darker and more opaque by adding more light. Therefore, an AR headset needs to be able to at least partially block light in order to make convincing images. It seems a lot easier just to go with Apple's approach of blocking all light rather than try to develop tech that will only selectively block the light behind the AR objects while allowing other light through. The blocking all light approach also allows you to hide other potential weaknesses of a device. For example, a lower field of view is much more distracting in a pass-through AR device as you still have your full peripheral vision. VR devices will generally black out the light outside of the FOV making it easier to ignore.
- ddalex 3y ago
  > you can't make something darker and more opaque by adding more light I mean... theoretically you can if you can match photon for photon with the inverse phase. Difficult, I know :)
AndrewKemendo 3y ago
Off the top of my head: 1. Field of view is limited to existing optics miniaturization 2. Subtractive shading (rendering black) might not be solvable 3. Variable focus objects in the same scene requires projecting n>2 significantly different wavefronts - not solved how to do this with a single vibrating element
- pornel 3y ago
  Do you know why an LCD panel can't be used to block light?
  - Tarragon 3y ago

CharlesW 3y ago

(Caveat: I don't know a lot about pass-through HMD AR, and I assume you're incredibly smart and that the patent is innovative.) > …it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology… I understand the framing as "compensatory technology", but is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. But a 100% digital vision pipeline seems like it could unlock interesting capabilities like "night vision", new ways of highlighting interesting objects, etc.

gjsman-1000 3y ago
I think the most prominent attempt at see-through AR was the Microsoft HoloLens. But if you've actually tried the HoloLens, the Field of View is atrociously, tragically small. The first HoloLens had a field of view of 30°*17.5°. The second HoloLens improved to 43°*29°, but it's still best described as "cramped." Couple that with almost all of the compute budget for the device going into vision processing and having very little compute left for actually running apps (the first HoloLens having a 1Ghz Intel Atom from 2015, the second a superior... Snapdragon 850). The other problem, of course, is that nothing can be truly solidly-colored. Everything has some opacity - which, combined with the FOV issue, is why HoloLens was never marketed as having anything to do with VR.
- moron4hire 3y ago
  The vision processing was done with a separate SoC. Your applications were not contending for compute time with the vision processing. That's just how anemic the CPU was.
- dfsl 3y ago
  [dead]
AndrewKemendo 3y ago
In theory the wavefront emulator is the simplest display - all of the "pixels" are rendered in your brain so you don't have to build an actual "display" HOWEVER if someone can get the input -> photon production pipeline to be less than ~10ms, then that does solve a lot of the rendering issues, however it doesn't solve all of the other long term problems that come with that amount of hardware - including weight and complexity. That said, there's a lot of known-unknowns that need to be solved, for example I don't have a solution for micropiezo resonance issues that I'm sure will crop up.