6 ms·

>It’s worth noting that Raja Koduri, SVP of AMD’s Radeon Technology Group, has stated that VR needs 16K per-eye at 144 Hz to emulate the human experience This

>It’s worth noting that Raja Koduri, SVP of AMD’s Radeon Technology Group, has stated that VR needs 16K per-eye at 144 Hz to emulate the human experience This is tangential to this thread, but can anyone in here explain the state of the art in eye tracking? Actually rendering 16k quality for my peripheral vision seems insane to me, so I'm really interested in the barriers between today's tech and a good foveated headset.

mastax 10y ago
Every year at CES, e3, etc. there are vendors with pretty good eye tracking headsets. APIs for foveated rendering are in place already. I'd imagine it'll be in the next generation of headsets.
- yourapostasy 10y ago
  I didn't think the civilian state of the art in foveated rendering is sufficiently good enough to fool our vision processing into believing it is seeing a "real" view, even in our peripheral vision, without head fatigue after say, 4 hours continuous usage, but I don't follow the field that closely because a couple years ago I thought that refinement level was so far away. A quick Google seems to reinforce that perception. I only hear PR puff pieces of military-grade gear that good now, like the F-35 helmet that is supposed to be light years ahead of anything civilians can get their hands on. Can anyone in the field give a précis on what is the state of the art?
VikingCoder 10y ago
Gee, and I always thought 15.5K and 143 Hz was enough.
- dsr_ 10y ago
  Well... if you have 20/20 vision, you can resolve about one arc-minute. (That's at your fovea. Everywhere else, not so great. But we will probably find it easier to render everything than track your eyei saccades and re-render fast enough to fool them.) Your field of vision is about 180 degrees wide (not really) and about 135 degrees vertically (full eye movement). So: 60 arc-minutes to a degree, 180 wide: 10800 pixels. 60 * 135 wide: 8100 pixels. An 8192x4096 is not there, but a 16384x8192 is. Now double that so you can get stereo vision (though really there's a lot of overlap that's going to be hard to arrange), update it 100 times a second in 48 bit color... 420 billion bits per second. Double that if you believe that we need 200Hz updates.
  - Pxtl 10y ago
    Yes, but shouldn't we eventually be eye-tracking anyways for depth-of-field effects? If you're eye-tracking already, you can leverage that for resolution optimization.
    - dpark 10y ago
      You still need the display to support the resolution even if you can optimize rendering costs.
    - the8472 10y ago
      To allow the eyes to actually refocus based on distance you want light field displays in your VR headsets
  - acchow 10y ago
    How many years away are we from being able to push this kind of throughput through a GPU?
    - paulmd 10y ago
      Oh boy, napkin math! (not sarcastic at all - I think this kind of ballpark estimate is actually super fun) A Titan XP/1080 Ti is finally just about enough to push 4K/60Hz with a single card. 16K would be 16 times as many pixels per frame, and 144 Hz is more than twice the refresh rate. To put that another way, we haven't even started to think about the connector standard that you could gang together to push that many pixels, let alone having a GPU that could actually push them. At this point you are pretty much talking about some kind of lossy compression being involved ("visually lossless" sure is a great euphemism). With lossy compression you might be able to get away with ganging together a couple of whatever DisplayPort 1.5 or 1.6 end up being. In fact with the end of Dennard scaling there's some fundamental problems with how you would even use a GPU like that as a consumer. Your average (US) household circuit is 120V/15A peak, and can be run at 80% load continuous (12A), which works out to 1440 watts at the wall. With an 80% efficient PSU that works out to 1152 watts inside the case. A Titan XP/1080 Ti pulls 270 watts at stock clocks [0]. So hypothetically even if you stacked four of them on an interposer you're now pulling 1080 watts inside the case, which is almost your entire circuit capacity. So by a naive calculation (16 times the pixels twice as fast = 32x divided by four GPU dies) we need at least an 8x improvement in overall efficiency before this is viable. (Four big GP102-sized chips on an interposer is technically possible right now - they only need to be on the interposer where you need interconnect bumps - they can hang off onto a support substrate, and you can have shared memory controllers/HBM2 stacks/etc on the interposer that make this appear to be one big GPU chip instead of SLI/Crossfire. Since the interposer is actually a chip all on its own, some of these auxiliary functions can actually be built into the interposer itself (the "active interposer" concept), with the biggest obstacle being getting the heat out since the interposer has other chips stacked on top of it...) 220V users obviously have things a little easier here (~twice the capacity per circuit). This cuts the overall efficiency improvement necessary down to 4x. There will also be some efficiency improvements on the software side. For starters we can cut quality a bit (that's 4K/60fps ultra), foveate rendering, and other software magic. Speeds don't scale perfectly negatively with increased resolution so you will get some savings there. Assume some speedup from DX12/Vulkan too. Oh and since this is 16K/144 Hz per eye I guess there's an implicit assumption here that we can make NVIDIA-style Multi Viewport Rendering work at near-100% efficiency, otherwise that's another factor of (up to) 2x that needs to be accounted for. So let's say that we need a 4x improvement in overall hardware/software efficiency. Let's say, something like a doubling in GPU throughput-per-watt and to double software efficiency before an absolutely state-of-the-art rig could even feasibly do this task on its own dedicated 220V/15A circuit. And there's a few fairly optimistic assumptions built into that 4x number. Take your best shot at how long it will take to quadruple efficiency in the post-Dennard era. Let's say 20-30 years, unless there's a massive breakthrough in materials science or optical computing or something. [0] https://www.techpowerup.com/reviews/NVIDIA/GeForce_GTX_1080_Ti/28.html https://www.techpowerup.com/reviews/NVIDIA/GeForce_GTX_1080_...
chairmanwow 10y ago
I cannot possible believe that rendering at this resolution is necessary. Most VR users don't turn their eyeballs while wearing headsets, but opt to use controls to adjust their perspective. I've read several accounts of hacks developers have taken when rendering VR games, but one that has kept cropping up is reducing resolution of areas in the periphery. While you certainly could use 16K displays for VR, I think it would be entirely unnecessary.
- intrasight 10y ago
  It needs 10K but I guess that becomes 16K due to base2 process.
- drewrv 10y ago
  Most VR users don't turn their eyeballs right now because the field of view is pretty small.
- IshKebab 10y ago
  > Most VR users don't turn their eyeballs while wearing headsets Err what? Of course they do!
PeterisP 10y ago
You may not need to render that pixel density in all places, but since you may need that in any place, your display hardware needs to be able to show that detail in any place where the user can point their eyeballs.
andmarios 10y ago
I think he may meant that if you were to process everything for a 360° view, you would need 16K. For games that are rendered on the fly, you probably can do with less (have the card render only the portion of the scene you see), but for video, you would have to shoot at 16K.
jl6 10y ago
Didn't someone once say 640K ought to be enough for anyone?

mastax 10y ago

Every year at CES, e3, etc. there are vendors with pretty good eye tracking headsets. APIs for foveated rendering are in place already. I'd imagine it'll be in the next generation of headsets.

yourapostasy 10y ago
I didn't think the civilian state of the art in foveated rendering is sufficiently good enough to fool our vision processing into believing it is seeing a "real" view, even in our peripheral vision, without head fatigue after say, 4 hours continuous usage, but I don't follow the field that closely because a couple years ago I thought that refinement level was so far away. A quick Google seems to reinforce that perception. I only hear PR puff pieces of military-grade gear that good now, like the F-35 helmet that is supposed to be light years ahead of anything civilians can get their hands on. Can anyone in the field give a précis on what is the state of the art?

VikingCoder 10y ago

Gee, and I always thought 15.5K and 143 Hz was enough.

dsr_ 10y ago
Well... if you have 20/20 vision, you can resolve about one arc-minute. (That's at your fovea. Everywhere else, not so great. But we will probably find it easier to render everything than track your eyei saccades and re-render fast enough to fool them.) Your field of vision is about 180 degrees wide (not really) and about 135 degrees vertically (full eye movement). So: 60 arc-minutes to a degree, 180 wide: 10800 pixels. 60 * 135 wide: 8100 pixels. An 8192x4096 is not there, but a 16384x8192 is. Now double that so you can get stereo vision (though really there's a lot of overlap that's going to be hard to arrange), update it 100 times a second in 48 bit color... 420 billion bits per second. Double that if you believe that we need 200Hz updates.
- Pxtl 10y ago
  Yes, but shouldn't we eventually be eye-tracking anyways for depth-of-field effects? If you're eye-tracking already, you can leverage that for resolution optimization.
  - dpark 10y ago
    You still need the display to support the resolution even if you can optimize rendering costs.
  - the8472 10y ago
    To allow the eyes to actually refocus based on distance you want light field displays in your VR headsets
- acchow 10y ago
  How many years away are we from being able to push this kind of throughput through a GPU?
  - paulmd 10y ago
    Oh boy, napkin math! (not sarcastic at all - I think this kind of ballpark estimate is actually super fun) A Titan XP/1080 Ti is finally just about enough to push 4K/60Hz with a single card. 16K would be 16 times as many pixels per frame, and 144 Hz is more than twice the refresh rate. To put that another way, we haven't even started to think about the connector standard that you could gang together to push that many pixels, let alone having a GPU that could actually push them. At this point you are pretty much talking about some kind of lossy compression being involved ("visually lossless" sure is a great euphemism). With lossy compression you might be able to get away with ganging together a couple of whatever DisplayPort 1.5 or 1.6 end up being. In fact with the end of Dennard scaling there's some fundamental problems with how you would even use a GPU like that as a consumer. Your average (US) household circuit is 120V/15A peak, and can be run at 80% load continuous (12A), which works out to 1440 watts at the wall. With an 80% efficient PSU that works out to 1152 watts inside the case. A Titan XP/1080 Ti pulls 270 watts at stock clocks [0]. So hypothetically even if you stacked four of them on an interposer you're now pulling 1080 watts inside the case, which is almost your entire circuit capacity. So by a naive calculation (16 times the pixels twice as fast = 32x divided by four GPU dies) we need at least an 8x improvement in overall efficiency before this is viable. (Four big GP102-sized chips on an interposer is technically possible right now - they only need to be on the interposer where you need interconnect bumps - they can hang off onto a support substrate, and you can have shared memory controllers/HBM2 stacks/etc on the interposer that make this appear to be one big GPU chip instead of SLI/Crossfire. Since the interposer is actually a chip all on its own, some of these auxiliary functions can actually be built into the interposer itself (the "active interposer" concept), with the biggest obstacle being getting the heat out since the interposer has other chips stacked on top of it...) 220V users obviously have things a little easier here (~twice the capacity per circuit). This cuts the overall efficiency improvement necessary down to 4x. There will also be some efficiency improvements on the software side. For starters we can cut quality a bit (that's 4K/60fps ultra), foveate rendering, and other software magic. Speeds don't scale perfectly negatively with increased resolution so you will get some savings there. Assume some speedup from DX12/Vulkan too. Oh and since this is 16K/144 Hz per eye I guess there's an implicit assumption here that we can make NVIDIA-style Multi Viewport Rendering work at near-100% efficiency, otherwise that's another factor of (up to) 2x that needs to be accounted for. So let's say that we need a 4x improvement in overall hardware/software efficiency. Let's say, something like a doubling in GPU throughput-per-watt and to double software efficiency before an absolutely state-of-the-art rig could even feasibly do this task on its own dedicated 220V/15A circuit. And there's a few fairly optimistic assumptions built into that 4x number. Take your best shot at how long it will take to quadruple efficiency in the post-Dennard era. Let's say 20-30 years, unless there's a massive breakthrough in materials science or optical computing or something. [0] https://www.techpowerup.com/reviews/NVIDIA/GeForce_GTX_1080_Ti/28.html https://www.techpowerup.com/reviews/NVIDIA/GeForce_GTX_1080_...

chairmanwow 10y ago

I cannot possible believe that rendering at this resolution is necessary. Most VR users don't turn their eyeballs while wearing headsets, but opt to use controls to adjust their perspective. I've read several accounts of hacks developers have taken when rendering VR games, but one that has kept cropping up is reducing resolution of areas in the periphery. While you certainly could use 16K displays for VR, I think it would be entirely unnecessary.

intrasight 10y ago
It needs 10K but I guess that becomes 16K due to base2 process.
drewrv 10y ago
Most VR users don't turn their eyeballs right now because the field of view is pretty small.
IshKebab 10y ago
> Most VR users don't turn their eyeballs while wearing headsets Err what? Of course they do!

PeterisP 10y ago

You may not need to render that pixel density in all places, but since you may need that in any place, your display hardware needs to be able to show that detail in any place where the user can point their eyeballs.

andmarios 10y ago

I think he may meant that if you were to process everything for a 360° view, you would need 16K. For games that are rendered on the fly, you probably can do with less (have the card render only the portion of the scene you see), but for video, you would have to shoot at 16K.

jl6 10y ago

Didn't someone once say 640K ought to be enough for anyone?