4 ms·

A field of view about 180 degrees wide, 135 degrees high, 1 arc minute in maximum pixel size, and updated 100 times per second… twice, to account for full stere

by dsr_ 6y ago

A field of view about 180 degrees wide, 135 degrees high, 1 arc minute in maximum pixel size, and updated 100 times per second… twice, to account for full stereography, and using 48 bits of color. 180 * 135 * 60 * 60 * 100 * 48 = 419904000000, 420 billion bits per second, without compression.

zamalek 6y ago
Keep in mind that the 90FPS benchmark is merely required to achieve "presence," which is the phenomenon of the brain accepting the virtual reality as reality. Higher framerates are always beneficial (as this also reduces latency). > 420Gbps I assume foveated compression (a pet idea of mine that I have no doubt someone else also has) is probably the only way to realistically deliver the stream to the HMD (assuming that some rendering pipeline that can deliver the uncompressed stream exists). You also have to be concerned about micro-saccades, which may increase the effective resolution of the eye.
- dsr_ 6y ago
  Right, 420Gb/s is the worst-case number. If you can track the fovea accurately, at 100 Hz you need to keep at least a 9 degree circle at full resolution -- more likely an 18 degree circle -- and even so, there's no mechanical system likely that's going to keep up with that motion, so you need a display screen at full resolution across the entire surface, even if you're only painting full resolution across a small sector each frame. Also also note: there's head motion to track and events to respond to. If you're playing a movie, that's pretty much your best case because you only have to pan and tilt a stream of things you already know about. Playing a game is medium bad, because you can predict the range of possible events. Doing AR mediation is worst, because you can't predict the rest of the universe.
- mparramon 6y ago
  When you say "foveated compression" do you mean >Apple will liberally use an already-known VR technique that involves using eye-tracking to render objects in the user's periphery at a lower fidelity than what the user is focusing on. ?
  - lgats 6y ago
    Interesting recent discussions on Foveated Rendering https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=foveated%20rendering&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - zamalek 6y ago
    No foveated compression is not foveated rendering. It is using the same general idea for a different area of the problem: a higher bitrate (on the wire, or across the wireless link) for in-fovea. Due to latency, you can't render the fovea area only; you need to cover quite a bit more in case the eye moves between submitting the command buffer and the final image being ready. Compression can happen much later in the process, where the latency is much more predictable or even fixed. This is useful because massive bandwidth is easiest along traces, harder along wires and hardest wirelessly.
thebean11 6y ago
What's the resolution in terms of a screen? Like height by width in pixels? Sorry if that's obvious from your comment, coming at this with just basic consumer knowledge about displays.
- dsr_ 6y ago
  The eye doesn't care about pixels, the eye cares about the smallest detail that it can distinguish. If you specify a particular distance you can figure out how small the pixels have to be at that range so that you can't see them. It's around one half of an arc-minute, which is to say, 1/120th of a degree. https://en.wikipedia.org/wiki/Visual_acuity https://en.wikipedia.org/wiki/Visual_acuity is a really useful overview. Now, we know that human eyes can only cover a little less than 180 degrees horizontally, and about 135 degrees vertically, so we could multiply each of those by 120 and get: 180 * 120 = 21600 across by 135 *120 = 16200 down. That's a screen where every pixel is guaranteed not to be distinguishable from the ones around it. If you had this on a hemispherical screen 10cm from your eye, your square pixels need to be no larger than 0.0145 mm on a diagonal, or if you prefer have a dot pitch of 0.0145 mm. Current state of commercial availability smartphone screens have dot pitches of about 0.045 mm, about 3 times too big. Does that help?
  - npunt 6y ago
    Probably can cut that down further due to : * People's vision getting worse with age * Each OLED pixel consists of 4 subpixels, so may only need to reach that acuity at sub pixel resolution * Acuity drops and eyestrain increases at the edges of vision so likely only need maybe 75% of our range U/D L/R at the highest resolution, and the outer edges can have much lower resolution Other hacks for foveated rendering include rendering peripheral vision at lower resolution and monochromatic, and not rendering in the blind spot, since our brains will just fill in those areas. May not be worth the complexity, but there's a lot of hacks that could take advantage of our visual system's quirks. I imagine ray tracing makes these hacks even easier to implement.
- trevortheblack 6y ago
  From the logic of the GP, that would be: Height: 135 * 60 (135 degrees at 60 pixels per degree) Width: 180 * 60 So 87 megapixels, or a 10.8K display with a 4:3 aspect ratio
mpoteat 6y ago
In the future there may be an integrated display technology that physically cannot display an arbitrary combination of pixels at high speed, but can only do so for the space of "images which humans are likely to see" which I suspect is an unimaginably smaller possibility space. I suppose something like a neural representation of a scene that is directly rendered on the underlying hardware, without first being converted to a raw bit stream.
- kridsdale1 6y ago
  This is an interesting return to the way that CRTs worked. They were technically 0-dimensional displays (a single beam changing in intensity, which was moved by a non-programmable magnet around the tube) In our modern future display, we would similarly blast data to only where it is needed at an instant in time.
red75prime 6y ago
Foveated imaging can dramatically decrease the required bandwidth. We have about 6 million cone cells and about 1 million of them is in the fovea centralis. Rods are more numerous, but they are mostly for low-light vision.

by dsr_ 6y ago

zamalek 6y ago

Keep in mind that the 90FPS benchmark is merely required to achieve "presence," which is the phenomenon of the brain accepting the virtual reality as reality. Higher framerates are always beneficial (as this also reduces latency). > 420Gbps I assume foveated compression (a pet idea of mine that I have no doubt someone else also has) is probably the only way to realistically deliver the stream to the HMD (assuming that some rendering pipeline that can deliver the uncompressed stream exists). You also have to be concerned about micro-saccades, which may increase the effective resolution of the eye.

dsr_ 6y ago
Right, 420Gb/s is the worst-case number. If you can track the fovea accurately, at 100 Hz you need to keep at least a 9 degree circle at full resolution -- more likely an 18 degree circle -- and even so, there's no mechanical system likely that's going to keep up with that motion, so you need a display screen at full resolution across the entire surface, even if you're only painting full resolution across a small sector each frame. Also also note: there's head motion to track and events to respond to. If you're playing a movie, that's pretty much your best case because you only have to pan and tilt a stream of things you already know about. Playing a game is medium bad, because you can predict the range of possible events. Doing AR mediation is worst, because you can't predict the rest of the universe.
mparramon 6y ago
When you say "foveated compression" do you mean >Apple will liberally use an already-known VR technique that involves using eye-tracking to render objects in the user's periphery at a lower fidelity than what the user is focusing on. ?
- lgats 6y ago
  Interesting recent discussions on Foveated Rendering https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=foveated%20rendering&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- zamalek 6y ago
  No foveated compression is not foveated rendering. It is using the same general idea for a different area of the problem: a higher bitrate (on the wire, or across the wireless link) for in-fovea. Due to latency, you can't render the fovea area only; you need to cover quite a bit more in case the eye moves between submitting the command buffer and the final image being ready. Compression can happen much later in the process, where the latency is much more predictable or even fixed. This is useful because massive bandwidth is easiest along traces, harder along wires and hardest wirelessly.

thebean11 6y ago

What's the resolution in terms of a screen? Like height by width in pixels? Sorry if that's obvious from your comment, coming at this with just basic consumer knowledge about displays.

dsr_ 6y ago
The eye doesn't care about pixels, the eye cares about the smallest detail that it can distinguish. If you specify a particular distance you can figure out how small the pixels have to be at that range so that you can't see them. It's around one half of an arc-minute, which is to say, 1/120th of a degree. https://en.wikipedia.org/wiki/Visual_acuity https://en.wikipedia.org/wiki/Visual_acuity is a really useful overview. Now, we know that human eyes can only cover a little less than 180 degrees horizontally, and about 135 degrees vertically, so we could multiply each of those by 120 and get: 180 * 120 = 21600 across by 135 *120 = 16200 down. That's a screen where every pixel is guaranteed not to be distinguishable from the ones around it. If you had this on a hemispherical screen 10cm from your eye, your square pixels need to be no larger than 0.0145 mm on a diagonal, or if you prefer have a dot pitch of 0.0145 mm. Current state of commercial availability smartphone screens have dot pitches of about 0.045 mm, about 3 times too big. Does that help?
- npunt 6y ago
  Probably can cut that down further due to : * People's vision getting worse with age * Each OLED pixel consists of 4 subpixels, so may only need to reach that acuity at sub pixel resolution * Acuity drops and eyestrain increases at the edges of vision so likely only need maybe 75% of our range U/D L/R at the highest resolution, and the outer edges can have much lower resolution Other hacks for foveated rendering include rendering peripheral vision at lower resolution and monochromatic, and not rendering in the blind spot, since our brains will just fill in those areas. May not be worth the complexity, but there's a lot of hacks that could take advantage of our visual system's quirks. I imagine ray tracing makes these hacks even easier to implement.
trevortheblack 6y ago
From the logic of the GP, that would be: Height: 135 * 60 (135 degrees at 60 pixels per degree) Width: 180 * 60 So 87 megapixels, or a 10.8K display with a 4:3 aspect ratio

mpoteat 6y ago

In the future there may be an integrated display technology that physically cannot display an arbitrary combination of pixels at high speed, but can only do so for the space of "images which humans are likely to see" which I suspect is an unimaginably smaller possibility space. I suppose something like a neural representation of a scene that is directly rendered on the underlying hardware, without first being converted to a raw bit stream.

kridsdale1 6y ago
This is an interesting return to the way that CRTs worked. They were technically 0-dimensional displays (a single beam changing in intensity, which was moved by a non-programmable magnet around the tube) In our modern future display, we would similarly blast data to only where it is needed at an instant in time.

red75prime 6y ago

Foveated imaging can dramatically decrease the required bandwidth. We have about 6 million cone cells and about 1 million of them is in the fovea centralis. Rods are more numerous, but they are mostly for low-light vision.