3 ms·

Streaming an arbitrary collection of windows instead of a single finished, composited framebuffer increases the bandwidth requirements by at least an order of m

Streaming an arbitrary collection of windows instead of a single finished, composited framebuffer increases the bandwidth requirements by at least an order of magnitude. That's never going to work well over WiFi.

ehsankia 3y ago
As long as the total number of pixels is less, I don't see what that has to be true, at least bandwidth wise? Compute wise, the vision might have to do slightly more to separate the buffer and composite them into the AR view in different places, but the bandwidth should be direction proportional to the number/size of each window. If I can fit all the windows on a 4K screen, then I don't see why the software can't split that and lay it out separate in my view instead of in a single rectangle.
- wtallis 3y ago
  It seems pretty obvious that nobody wants to be limited to the total number of pixels being merely equivalent to tiling across a 4k screen.
- saagarjha 3y ago
  Some of the windows will be obscured by others. If you stream a 100 windows to visionOS, it's possible to lay them out so that none of them cover each other and you have to render them all. On a flat screen there is a limit to how many pixels you need to paint.
jayd16 3y ago
Just compress the stream. Total pixels increase the vram on the device but popping out a static window shouldn't take any more than a trivial amount of streaming bandwidth.
- wtallis 3y ago
  Everything is already compressed. Uncompressed 1080p is 3Gbit/s, which is already well beyond what's actually achievable with WiFi. Allowing an unbounded number of window surfaces to be streamed invites people to use it in a way that current technology simply cannot provide the kind of slick experience that Apple needs from this product.
  - jayd16 3y ago
    Seeing as you have the eye tracking you could probably greatly reduce the FPS/bitrate of windows that are out of gaze. This kind of foveated rendering seems to already be used in their ecosystem so I assume its sufficiently slick for Apple. Apps will sleep when not being gazed and such. This allows you to restrict full bitrate to a single window with a maximum resolution and keep the user experience high and astonishment low.
- saagarjha 3y ago
  Keep in mind that you are popping out a static window 60 times a second or so.
vunderba 3y ago
Use the touted eye tracking feature and compress that particular applications stream inversely to how much the user is focusing on it within the Vision Pro.
- wtallis 3y ago
  Does foveated rendering work with the extra latency of a round trip over WiFi? I can certainly see this working fine on a head tracking scale, but I'm less sure about eye tracking. I also wonder if existing video compression hardware can actually make use of such outside feedback to adapt bitrate and quality with the necessary flexibility.

ehsankia 3y ago

As long as the total number of pixels is less, I don't see what that has to be true, at least bandwidth wise? Compute wise, the vision might have to do slightly more to separate the buffer and composite them into the AR view in different places, but the bandwidth should be direction proportional to the number/size of each window. If I can fit all the windows on a 4K screen, then I don't see why the software can't split that and lay it out separate in my view instead of in a single rectangle.

wtallis 3y ago
It seems pretty obvious that nobody wants to be limited to the total number of pixels being merely equivalent to tiling across a 4k screen.
saagarjha 3y ago
Some of the windows will be obscured by others. If you stream a 100 windows to visionOS, it's possible to lay them out so that none of them cover each other and you have to render them all. On a flat screen there is a limit to how many pixels you need to paint.

jayd16 3y ago

Just compress the stream. Total pixels increase the vram on the device but popping out a static window shouldn't take any more than a trivial amount of streaming bandwidth.

wtallis 3y ago
Everything is already compressed. Uncompressed 1080p is 3Gbit/s, which is already well beyond what's actually achievable with WiFi. Allowing an unbounded number of window surfaces to be streamed invites people to use it in a way that current technology simply cannot provide the kind of slick experience that Apple needs from this product.
- jayd16 3y ago
  Seeing as you have the eye tracking you could probably greatly reduce the FPS/bitrate of windows that are out of gaze. This kind of foveated rendering seems to already be used in their ecosystem so I assume its sufficiently slick for Apple. Apps will sleep when not being gazed and such. This allows you to restrict full bitrate to a single window with a maximum resolution and keep the user experience high and astonishment low.
saagarjha 3y ago
Keep in mind that you are popping out a static window 60 times a second or so.

vunderba 3y ago

Use the touted eye tracking feature and compress that particular applications stream inversely to how much the user is focusing on it within the Vision Pro.

wtallis 3y ago
Does foveated rendering work with the extra latency of a round trip over WiFi? I can certainly see this working fine on a head tracking scale, but I'm less sure about eye tracking. I also wonder if existing video compression hardware can actually make use of such outside feedback to adapt bitrate and quality with the necessary flexibility.