3 ms·
Tangentially related: I've recently been doing some VR work in Unreal Engine, and also reading Blindsight by Peter Watts, and this made me wonder: are saccades
by TeMPOraL 6y ago
Tangentially related: I've recently been doing some VR work in Unreal Engine, and also reading Blindsight by Peter Watts, and this made me wonder: are saccades truly random? Could they be made predictable through some special way of crafting a headset, or through some stimulation? If so, then perhaps one day we'll be able to render only the pixels the mind can actually perceive at any given moment.
- 082349872349872 6y agoNot prediction, but: there used to be an Exploratorium exhibit with a large screen which updated randomly. One could toggle between flashing (erase to black) updates and smooth (just overwritten) updates. The difference in sensation of change was palpable.
- kebman 6y agoAre you talking about Critical Flicker Fusion here?
- 082349872349872 6y agoPossibly. (I'm unfamiliar with the term, TIL!) As I dimly recall from last century, we are sensitive to flicker in the coarse-resolution field, but insensitive to non-flickering change outside the fine-resolution field.
- kebman 6y agoThen you're probably familiar with the 24 frames per second cinema standard for projecting films. Except each frame is actually projected three times, leading to a shutter speed of 72 frames per second. (In reality there are far more options today with IMAX and video tech, though.) Even though there's only 24 images on the film per second, this triple exposure results in a smoother experience for the viewer. This is due to how critical flicker fusion works (a.k.a. persistence of vision). [1] (Though another reason for doing it is simply so that the film won't be burnt, since those xenon gas projection lamps run very hot.) See also beta movement and phi phenomenon. [2] These physiological phenomena are of course very important to consider when making VR devices and games. In order to create the effect mechanically in the film projector, each frame has to be stationary before the shutter is opened. If not, all you'd see is a blur. Thus, when the film is pulled forward, the shutter blocks the light from projecting the image onto the screen. In fact, during some half of the movie, people are actually sitting is pitch black darkness. Think about that next time you go to the movies! ;) Please note that peripheral vision has a higher sensitivity to flicker than foveal vision. I'm unsure how interlacing affects that, though, if applickable. This effect might also be different on video systems where various forms of interlacing may or may not be used. [1]: https://en.wikipedia.org/wiki/Flicker_fusion_threshold https://en.wikipedia.org/wiki/Flicker_fusion_threshold [2]: https://en.wikipedia.org/wiki/Beta_movement https://en.wikipedia.org/wiki/Beta_movement (Please excuse my lack of sourcing. Most of this is off the back of my head, and from books I am no longer in posission of... But the Wikipedia links should give you a good start.)
- TeMPOraL 6y ago> Thus, when the film is pulled forward, the shutter blocks the light from projecting the image onto the screen. Ironically, it sounds like a complement to saccades! Where saccades involve shutting off vision when your eye moves, this involves shutting off projection when the image moves.
- lopmotr 6y agoWould eye tracking be fast enough to follow saccades and generate the high resolution parts on the fly? I assumed it would and we'll eventually have VR at resolution of human perception done that way. I hope so!
- TeMPOraL 6y agoEventually, maybe? But I was hoping there would be a way to cheat and somehow force saccades to happen on a pattern that can be predicted beforehand, so that we could render only the parts of the image that the eyes are just about to jump to.
- lopmotr 6y agoProbably to some extent. Movie makers anticipate what part of the screen viewers will look at. They can control it with content, motion, focus, etc. Maybe you're hoping for something more automatic? For a simple looping video we might get bored and our eyes start wandering around, but if we're paying attention to the content, we probably keep focused on the moving/important objects.
- TeMPOraL 6y agoYes, I'm hoping for something automatic enough to allow for a turbocharged version of Foveated Rendering. One where, instead being dead ahead of you, the region of high-resolution rendering would follow your eyes as they saccade - so you'd never be able to tell the headset renders only a tiny fraction of the image at any given time, while giving it even more time to produce a high-quality render there.
- mncharity 6y ago> Would eye tracking be fast enough to follow saccades MEMS kHz eye tracking enables even predicting where a saccade will land 20+ ms before: https://www.youtube.com/watch?v=JEg4l5KuQgI&t=452 https://www.youtube.com/watch?v=JEg4l5KuQgI&t=452 (AdHawk @ AWE 2018).
- 6y ago
- codeulike 6y agoWhat makes you think saccades are random? When you are reading text, for example, they are methodical. We have had machines that do what you describe for decades. Here's Daniel Dennett in Consciousness Explained (1992) When your eyes dart about in saccades, the muscular contractions that cause the eyeballs to rotate are ballistic actions: Your fixation points are unguided missiles whose trajectories at lift-off determine where and when they will hit ground zero at a new target ... Amazingly, a computer equipped with an automatic eye-tracker can detect and analyse the lift-off in the first few milliseconds of a saccade, calculate where ground zero will be, and before the saccade is over, erase the word on the screen at ground zero and replace it with a different word of the same length. What do you see? Just the new word, and with no sense at all of anything having been changed. As you peruse the text on the screen, it seems to you for all the world as stable as if the words were carved in marble, but to another person reading the same text over your shoulder (and saccading to a different drummer) the screen is aquiver with changes. The effect is overpowering. When I first encountered an eye-tracker experiment, and saw how oblivious subjects were (apparently) to the changes flickering on the screen, I asked if I could be a subject. I wanted to see for myself. I was seated at the apparatus, and my head was immobilized by having me bite on a "bite bar". This makes the job easier for the eye-tracker, which bounces an unoticeable beam of light off the lens of the subject's eye, and analyzes the return to detect any motion of the eye. While I waited for the experimenters to turn on the apparatus, I read the text on the screen. I waited, and waited, eager for the trails to begin. I got impatient. "Why dont you turn it on?" I asked. "It is on," they replied.
- TeMPOraL 6y ago> What makes you think saccades are random? I didn't mean "random" as in governed by an RNG, just "random" as in we can't obviously predict their pattern. The opposite of that would be being able to tell when they'll happen and where they'll end before they start, and/or make them happen on demand and land on desired target. > We have had machines that do what you describe for decades. I didn't knew that. Thanks for citation. This leads me to ask: so why aren't we employing this for VR?
- codeulike 6y ago
- RobotCaleb 6y agoOne of my favorite books. You might be interested in a fan movie launching next week. http://blindsight.space/ http://blindsight.space/ https://mailchi.mp/b4c8b26e025d/blindsight-project-update-september-2528877 https://mailchi.mp/b4c8b26e025d/blindsight-project-update-se...
- TeMPOraL 6y agoThanks, very timely! I'm in the middle of reading the sequel/sidequel, Echopraxia, right now.
- modeless 6y agoSaccades are neither random nor very predictable. They are voluntary. You can predict the "salient" parts of an image that humans will prefer to look at, and use that information to e.g. improve video compression. But your predictions will always be wrong sometimes because humans are unpredictable. It should be feasible to detect saccades instead of predicting them, and render new pixels more quickly than the eye can move. But it is right at the edge of what's possible, and really needs more reliable eye tracking than is generally available today, along with purpose built rendering techniques and higher resolution + higher field of view displays.