6 ms·

This is also the principle behind foveated rendering (https://en.wikipedia.org/wiki/Foveated_imaging https://en.wikipedia.org/wiki/Foveated_imaging) where you o

This is also the principle behind foveated rendering (https://en.wikipedia.org/wiki/Foveated_imaging https://en.wikipedia.org/wiki/Foveated_imaging) where you only render the graphics in high detail where the viewer is looking, and use a low-res image for the rest with the goal of saving computing power. Big area of development right now for VR!

Phemist 11y ago
That's really cool! I'm not 100% sure, but what I remember from reading a few psych papers about this is that as long as the "mask" is similar to the original in a few easy to calculate sampling statistics (colour distribution, overall hue), it doesn't actually need to look anything like the original image, even calculating low-res imagery shouldn't be necessary. Our peripheral vision is just that bad (at least for static imagery, movement detection is actually pretty good). The foveated part is where the magic happens, and we've gotten pretty good at fooling ourself into believing this "foveated" part is much larger than it actually is.
- kedean 11y ago
  Hey, something I know about! I actually worked on one of those research projects in college, programming the experiments. The idea is that there is a certain radius around the focal point where you stop being able to detect changes. I'm not sure what the final results were, but the theory was that you can calculate how blurry an image can be and still be discernable based on how far it is from the focal point of your vision. It was surprising how good people are at detecting changes in a blurry picture that's way out in their periphery.
  - Phemist 11y ago
    My prof used to joke that, whenever he'd discuss fovea and periphery, that if we had a fovea the size of our full field of view, we'd need a brain the size of an elephant to process all that information. It's interesting how we're very sensitive to sudden changes (thus movement) in our periphery, but are so bad at classifying/identifying static imagery. I remember reading about right eye and left eye dominance. Where they'd keep an image on the screen saccade invariant (ie, compensate for any saccades that were made). Slowing moving a letter/character/word/whatever to the edges of the participant's field of view and asking when the character was no longer legible. This happened surprisingly quickly, but at different positions for the left eye and right eye for pretty much all participants..
chestervonwinch 11y ago
I've seen this idea used in machine learning as well. There was a paper [1] whose goal was a binary classification of the center pixel in a region of interest. Interestingly, their results improved when they applied a foveal blurring surrounding the pixel to be classified. [1]: http://people.idsia.ch/~juergen/nips2012.pdf http://people.idsia.ch/~juergen/nips2012.pdf
- Phemist 11y ago
  Interesting! Thanks for sharing! There are algorithms that mimic fixation paths the eyes follow when presented with a novel image, very much related to modeling dopaminergic systems. It seemed to find task-relevant information dense areas first, and then slowly spread out to less information dense areas. I wonder if there'd be any benefit to running these algorithms on images, basically turning them into a video, and then running classifiers based on this video (with or without foveal blurring
  - otakucode 11y ago
    Do you have any more information about these algorithms? Or perhaps links to a paper or two? It gives me a couple ideas and sounds very interesting! I do wonder as you do as well. Feeding an image to a neural network instead of as a single input, but as a series of inputs separated over time, 'reusing' the same neurons for different portions of the image might allow for interesting feedback to develop.
    - Phemist 11y ago
      Apparently I can reply now, repost just so it shows up in your comment thread I remember seeing a video of a robot that would attend to different parts of a scene based on the "saliency", in the sense of novelty, of its features. I can't find the specific video, but I think the model running the robot is related to: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3930917/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3930917/ The paper is quite dense, as it describes biologically plausible models of dopaminergic systems, which makes the model quite complex as well, but it's interesting because this system is considered quite 'low-level', no cortex involved. I'll add more if I can find any..
soared 11y ago
I just posted this in another comment, but M$ research has awesome foveated rendering: http://research.microsoft.com/apps/pubs/default.aspx?id=176610 http://research.microsoft.com/apps/pubs/default.aspx?id=1766...
- martinpw 11y ago
  I got a live demo of this at a conference a few years back. There was a line to see the demo, so I first waited while others did it. They first showed the full resolution image, then enabled the foveated rendering. The latter looked terrible - a tiny patch of hires imagery and a massive blur for everything else. When it was my turn, they started with the full resolution image. I was waiting for them to enable the foveated rendering - until they told me it was already running. I could not tell at all that it was not all rendering at full resolution. Really impressive. The main presenter mentioned that the optic nerve/brain processing 'shuts down' for up to 40ms during a saccade, so they have that long to render the small region at full resolution between the time when your eye has its new target and the optic nerve comes back on line.
  - soared 11y ago
    I'm so glad to hear this! Do you know of any progress since then? That very closely reminds me of my first time trying VR (last week). I sat and watched on the 2D monitor someone else doing it, and thought it looked incredibly stupid. Then I put on the headset..
    - martinpw 11y ago
      I don't know of any explicit details of progress, but everyone I've mentioned it to in the VR space nods knowingly when I mention it, so I take that to mean that it is an area of active research and development.
shurcooL 11y ago
Real-time raytracing would benefit so much from that.
TrevorJ 11y ago
I am pretty excited about this, I could see this being used with an variant of DLP technology to deliver ultra-high resolution and pixel density only to the area of screen you are looking at.
bigiain 11y ago
Heh - I remember back in the very late '80s or early '90s, my dad was working on the flight simulator for the "swing wing" F111E's new avionics package that Australia was getting. The sim cockpit had a pair of Silicon Graphic Reality Engine^2s - one driving a projector that lit up a 5 or 6 meter diameter 1/4 spherical screen at low-ish resolution, and another driving a projector that was aimable and slaved to the pilot's helmet (and maybe even eye tracking, I can't remember) that projected a small patch of high resolution imagery exactly where the pilot was looking. If you knew what it was doing, it was easy enough to "catch out" that system and see the edges where the two images joined, but once immersed in flying it disappeared completely. It was spectacularly obvious what was going on if you were watching the screen while someone else had the helmet on. I _so_ wanted one of those Reality Engines back then, I suspect my phone now has more graphics processing power though (I'm pretty sure my Galaxy S6 in a Gear VR does a significantly better job than that multi million dollar military project ~20 years ago...)
- derefr 11y ago
  The SGI "Reality Engine" was also the GPU that ended up in the Nintendo 64, wasn't it? How much more powerful was the RE2?
derefr 11y ago
It's always seemed to me that we could get much higher-quality VR if we could manage to set up a foveated display: a high-DPI display embedded in concentric rings of progressively-lower-DPI displays. It's easy enough to get quality-control clamped for a 600DPI display if you only have to make them a square inch in size; and your memory bandwidth and parallel processing needs go way down if the outside rings can actually be treated as a small screen, rather than as an extremely-high-resolution screen displaying a (monotonous and blurry) image. Of course, the big obvious problem with such a display is that the eye moves, and moves faster than you could possibly move around the display. The real key, I would think, would be something equivalent to a metamaterial convex lens, that could be "tilted" and "flexed" in the same way the lenses in our own eyes can, to redirect and "refocus" the centre of the image to the new eye position without actually moving it per se. We already have a technology to achieve this sort of "tilting" and "flexing", it turns out: magnetic deflection, as seen in CRTs. No reason you couldn't use it to deflect a continuous parallel matrix of rays by a constant amount, rather than one continuously-shifting beam. Heck, you could use an array of coherent emitters (laser diodes) rather than point-source diodes, and use phosphorous on the intermediary panel like the good old days.

Phemist 11y ago

That's really cool! I'm not 100% sure, but what I remember from reading a few psych papers about this is that as long as the "mask" is similar to the original in a few easy to calculate sampling statistics (colour distribution, overall hue), it doesn't actually need to look anything like the original image, even calculating low-res imagery shouldn't be necessary. Our peripheral vision is just that bad (at least for static imagery, movement detection is actually pretty good). The foveated part is where the magic happens, and we've gotten pretty good at fooling ourself into believing this "foveated" part is much larger than it actually is.

kedean 11y ago
Hey, something I know about! I actually worked on one of those research projects in college, programming the experiments. The idea is that there is a certain radius around the focal point where you stop being able to detect changes. I'm not sure what the final results were, but the theory was that you can calculate how blurry an image can be and still be discernable based on how far it is from the focal point of your vision. It was surprising how good people are at detecting changes in a blurry picture that's way out in their periphery.
- Phemist 11y ago
  My prof used to joke that, whenever he'd discuss fovea and periphery, that if we had a fovea the size of our full field of view, we'd need a brain the size of an elephant to process all that information. It's interesting how we're very sensitive to sudden changes (thus movement) in our periphery, but are so bad at classifying/identifying static imagery. I remember reading about right eye and left eye dominance. Where they'd keep an image on the screen saccade invariant (ie, compensate for any saccades that were made). Slowing moving a letter/character/word/whatever to the edges of the participant's field of view and asking when the character was no longer legible. This happened surprisingly quickly, but at different positions for the left eye and right eye for pretty much all participants..

chestervonwinch 11y ago

I've seen this idea used in machine learning as well. There was a paper [1] whose goal was a binary classification of the center pixel in a region of interest. Interestingly, their results improved when they applied a foveal blurring surrounding the pixel to be classified. [1]: http://people.idsia.ch/~juergen/nips2012.pdf http://people.idsia.ch/~juergen/nips2012.pdf

Phemist 11y ago
Interesting! Thanks for sharing! There are algorithms that mimic fixation paths the eyes follow when presented with a novel image, very much related to modeling dopaminergic systems. It seemed to find task-relevant information dense areas first, and then slowly spread out to less information dense areas. I wonder if there'd be any benefit to running these algorithms on images, basically turning them into a video, and then running classifiers based on this video (with or without foveal blurring
- otakucode 11y ago
  Do you have any more information about these algorithms? Or perhaps links to a paper or two? It gives me a couple ideas and sounds very interesting! I do wonder as you do as well. Feeding an image to a neural network instead of as a single input, but as a series of inputs separated over time, 'reusing' the same neurons for different portions of the image might allow for interesting feedback to develop.
  - Phemist 11y ago
    Apparently I can reply now, repost just so it shows up in your comment thread I remember seeing a video of a robot that would attend to different parts of a scene based on the "saliency", in the sense of novelty, of its features. I can't find the specific video, but I think the model running the robot is related to: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3930917/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3930917/ The paper is quite dense, as it describes biologically plausible models of dopaminergic systems, which makes the model quite complex as well, but it's interesting because this system is considered quite 'low-level', no cortex involved. I'll add more if I can find any..

soared 11y ago

I just posted this in another comment, but M$ research has awesome foveated rendering: http://research.microsoft.com/apps/pubs/default.aspx?id=176610 http://research.microsoft.com/apps/pubs/default.aspx?id=1766...

martinpw 11y ago
I got a live demo of this at a conference a few years back. There was a line to see the demo, so I first waited while others did it. They first showed the full resolution image, then enabled the foveated rendering. The latter looked terrible - a tiny patch of hires imagery and a massive blur for everything else. When it was my turn, they started with the full resolution image. I was waiting for them to enable the foveated rendering - until they told me it was already running. I could not tell at all that it was not all rendering at full resolution. Really impressive. The main presenter mentioned that the optic nerve/brain processing 'shuts down' for up to 40ms during a saccade, so they have that long to render the small region at full resolution between the time when your eye has its new target and the optic nerve comes back on line.
- soared 11y ago
  I'm so glad to hear this! Do you know of any progress since then? That very closely reminds me of my first time trying VR (last week). I sat and watched on the 2D monitor someone else doing it, and thought it looked incredibly stupid. Then I put on the headset..
  - martinpw 11y ago
    I don't know of any explicit details of progress, but everyone I've mentioned it to in the VR space nods knowingly when I mention it, so I take that to mean that it is an area of active research and development.

shurcooL 11y ago

Real-time raytracing would benefit so much from that.

TrevorJ 11y ago

I am pretty excited about this, I could see this being used with an variant of DLP technology to deliver ultra-high resolution and pixel density only to the area of screen you are looking at.

bigiain 11y ago

Heh - I remember back in the very late '80s or early '90s, my dad was working on the flight simulator for the "swing wing" F111E's new avionics package that Australia was getting. The sim cockpit had a pair of Silicon Graphic Reality Engine^2s - one driving a projector that lit up a 5 or 6 meter diameter 1/4 spherical screen at low-ish resolution, and another driving a projector that was aimable and slaved to the pilot's helmet (and maybe even eye tracking, I can't remember) that projected a small patch of high resolution imagery exactly where the pilot was looking. If you knew what it was doing, it was easy enough to "catch out" that system and see the edges where the two images joined, but once immersed in flying it disappeared completely. It was spectacularly obvious what was going on if you were watching the screen while someone else had the helmet on. I _so_ wanted one of those Reality Engines back then, I suspect my phone now has more graphics processing power though (I'm pretty sure my Galaxy S6 in a Gear VR does a significantly better job than that multi million dollar military project ~20 years ago...)

derefr 11y ago
The SGI "Reality Engine" was also the GPU that ended up in the Nintendo 64, wasn't it? How much more powerful was the RE2?

derefr 11y ago

It's always seemed to me that we could get much higher-quality VR if we could manage to set up a foveated display: a high-DPI display embedded in concentric rings of progressively-lower-DPI displays. It's easy enough to get quality-control clamped for a 600DPI display if you only have to make them a square inch in size; and your memory bandwidth and parallel processing needs go way down if the outside rings can actually be treated as a small screen, rather than as an extremely-high-resolution screen displaying a (monotonous and blurry) image. Of course, the big obvious problem with such a display is that the eye moves, and moves faster than you could possibly move around the display. The real key, I would think, would be something equivalent to a metamaterial convex lens, that could be "tilted" and "flexed" in the same way the lenses in our own eyes can, to redirect and "refocus" the centre of the image to the new eye position without actually moving it per se. We already have a technology to achieve this sort of "tilting" and "flexing", it turns out: magnetic deflection, as seen in CRTs. No reason you couldn't use it to deflect a continuous parallel matrix of rays by a constant amount, rather than one continuously-shifting beam. Heck, you could use an array of coherent emitters (laser diodes) rather than point-source diodes, and use phosphorous on the intermediary panel like the good old days.