4 ms·

What does the Apple Vision Pro offer in terms of expanded vision? Can I zoom in on something while wearing them? Can I use them to see into the Infrared spect

What does the Apple Vision Pro offer in terms of expanded vision? Can I zoom in on something while wearing them? Can I use them to see into the Infrared spectrum? What about decoding QR Codes in real time and displaying the metadata? One use case for me would probably be to wear them around the house (or at the grocery) and expand the text on labels. Being over 50, my presbyopia forces me to constantly have my glasses available. If I used AVP I wouldn't need glasses, right? What about hooking it up to my Tesla so that I can drive with them on and not have to look down to see my current speed? Talk about a HUD! Or what about expanding our natural capabilities? FLIR cameras are getting pretty reasonable; could there be an add-on for detecting heat patterns? Or what about visualizing ultrasonic waves; like in "The Dark Knight"?

zarzavat 3y ago
I imagine the way these work is that there’s a fast hardware path from camera to display, with the OS alpha composited on top. Any kind of post-processing of the pass-through video will introduce latency. So my guess would be no zooming.
- dTal 3y ago
  I doubt it - the cameras are not at eye location. I expect they employ a similar solution to the Quest, which has no front-facing cameras at all - it builds a 3d model of the scene on the fly using photogrammetry on the corner cameras (if Apple do this they'd avail themselves of the built in LIDAR) and folds it into the rest of the rendering stack. This has theoretically better latency for the important part (head movement) than piping cameras directly into the displays, because it's perfectly synchronized with the display refresh without having to wait for a camera to complete a frame. I expect Apple does something similar.
  - zaptrem 3y ago
    This isn't what Quest does. Photogrammetry isn't that good (nor that fast). They just warp the images from the down/outward-facing cameras.
    - dTal 3y ago
      Well sort of. It's a depth-accurate warp, fed from the SLAM system. This is functionally equivalent to texturing a depth map with the cameras. It has to be this way because there's no other way to simulate having a camera in a different spot than it really is. If they were simple 2d warps, parallax, occlusion, and stereopsis would be all wrong, and you'd feel violently ill any time you moved your head. You can see the system break down if you bring your hand too close to the headset. You're certainly not getting raw passthrough images. I'm not intimately familiar with the precise technical details of the Quest system, but I do know that real time photogrammetry even more sophisticated than depth maps can be achieved on mobile chipsets - google LSD-SLAM.
      - zaptrem 3y ago
        This makes a lot more sense, thanks for elaborating.

zarzavat 3y ago

I imagine the way these work is that there’s a fast hardware path from camera to display, with the OS alpha composited on top. Any kind of post-processing of the pass-through video will introduce latency. So my guess would be no zooming.

dTal 3y ago
I doubt it - the cameras are not at eye location. I expect they employ a similar solution to the Quest, which has no front-facing cameras at all - it builds a 3d model of the scene on the fly using photogrammetry on the corner cameras (if Apple do this they'd avail themselves of the built in LIDAR) and folds it into the rest of the rendering stack. This has theoretically better latency for the important part (head movement) than piping cameras directly into the displays, because it's perfectly synchronized with the display refresh without having to wait for a camera to complete a frame. I expect Apple does something similar.
- zaptrem 3y ago
  This isn't what Quest does. Photogrammetry isn't that good (nor that fast). They just warp the images from the down/outward-facing cameras.
  - dTal 3y ago
    Well sort of. It's a depth-accurate warp, fed from the SLAM system. This is functionally equivalent to texturing a depth map with the cameras. It has to be this way because there's no other way to simulate having a camera in a different spot than it really is. If they were simple 2d warps, parallax, occlusion, and stereopsis would be all wrong, and you'd feel violently ill any time you moved your head. You can see the system break down if you bring your hand too close to the headset. You're certainly not getting raw passthrough images. I'm not intimately familiar with the precise technical details of the Quest system, but I do know that real time photogrammetry even more sophisticated than depth maps can be achieved on mobile chipsets - google LSD-SLAM.
    - zaptrem 3y ago
      This makes a lot more sense, thanks for elaborating.