10 ms·

Spreadsheet "breaks" Apple Vision Pro eye-tracking

146 points by Topfi 3y ago

jwalton 3y ago
I cannot be the only one who reads AVP as “Alien vs. Predator”.
orenlindsey 3y ago
To be fair, this is a 1.0 version of brand new software and hardware. By the time we get to visionOS 3.0, and maybe even before that, this will be gone.
- sneak 3y ago
  I don’t think this will ever improve significantly (maybe a little bit with software tweaks) on AVP1 hardware. The current algorithm is not terrible; it may be possible to improve the artifacting a bit but ultimately the rendering setup exists to cope with the resolution of the displays. The state of the art in cost-effective HMDs will need to advance a bit for this to really get “fixed”, as TFA notes. It is telling that it took a very contrived test (a white grid of fine lines on black) for this to be obvious. Most things displayed by most users won’t run into this problem. I agree that this thing is not yet monitor-replacement-level yet. I mostly just want it for watching movies in bed or on planes.
- MBCook 3y ago
  All the “it’s not good enough” posts amaze me. As you said we’re still on 1.0. It hasn’t even been a week. I bet Apple can improve it without further hardware, let alone future versions. However I was wondering just how accurate this could get yesterday. Because of saccades the eye never really stops and just looks at one thing right? Which would mean we can never tell what someone is looking at, only the area of the eye is heavily focused around. Sort of a probability field. Just how small can we make that field and have it still feel accurate? Or does this introduce something of the “thumb“ problem on touchscreens where you have to just do your best to interpret what the user probably wanted?
  - deleted 3y ago
    [deleted]
  - crooked-v 3y ago
    So far with mine, it feels like fingertip size is about the smallest 'eye target' I can use for a close-up screen (think roughly the position of a held iPad) and still have 100% accuracy. I would bet that the limits of eye accuracy are the math basis of window resizing with push/pull movement, so that the accuracy rate remains the same for farther-away virtual screens.
  - rozab 3y ago
    I believe eye tracking systems have been used for weapons targeting in F-35s for about 10 years. They wouldn't be using it for that usecase if it wasn't pretty accurate
    - zarzavat 3y ago
      If you’re in a fighter jet and there’s an enemy aircraft you don’t need much encouragement to look at it, whereas with computers people look away from the UI element they are interacting with all the time, for example I’m typing on the iOS keyboard now but looking at the text. Also the number of objects in a fighter jet is going to be rather small so the radius for selection around each object can be large.
    - MBCook 3y ago
      True. But an F-35 cockpit has more space for cameras/etc and a slightly higher price. I also suspect selecting from very close together (in your field of view) enemies isn’t a too common. On the other hand on a site like HN there are a ton of very small targets right next to eachother.
- _ugfj 3y ago
  Let's hope we never do. Of course, there's very little hope but we all would be better off if this product didn't exist. Let me quote Paris Marx's excellent summary from his Apple Vision Pro preview https://disconnect.blog/apples-vision-pro-headset-deserves/ https://disconnect.blog/apples-vision-pro-headset-deserves/ > Tech companies want us isolated and constantly staring at screens because it drives profit and of course > The Vision Pro lets users further separate themselves from their surroundings and even create “digital personas” for video calls so other people are talking to a digital, abstracted version of themselves instead of seeing the real person. Even The Verge acknowledges: > using the Vision Pro is such a lonely experience, regardless of the weird ghost eyes on the front. Of course it's Apple so it'll catch on and what in 1969 was predicted to happen in 4545 will happen, well, quite a few years earlier : In the year 4545 ... : Nobody's gonna look at you And of course https://img.ifunny.co/images/3461f2f61d0ac8385d0c1985ac415547f6a1ad99783efc91a20dbe94d4158315_1.jpg https://img.ifunny.co/images/3461f2f61d0ac8385d0c1985ac41554...
  - CatWChainsaw 3y ago
    We can always hope that Vision Pro turns into Glasshole 2.0.
threeseed 3y ago
This is simply how foveated rendering works. Nothing is broken here. It's an optimisation technique that is needed now because we don't have the resources to handle the full resolution. But this will change over the coming years. For now maybe just increase the font size on the spreadsheet.
- crooked-v 3y ago
  The "broken" part is, outside of the visible difference in the foveation, the "fizzing" effect from whatever's going on with the antialiasing being weird with the zoomed-out grid.
  - kevingadd 3y ago
    I think these are typically called moire patterns, and you will see them almost any time a high frequency signal is being rendered at a lower frequency. Anti-aliasing, box filters etc are common workarounds for them but you can't always make them completely go away.
Havoc 3y ago
Not sure it matters at this stage? One day excel gang will no doubt be using this type of VR setup. It makes for a very poor early adopter crowd though - Excel users need keyboards not current finger gesture setup. They're very reliant on muscle memory, shortcuts and predictable low latency interactions with literally a single piece of software. Unless you want to pay your accountant twice as much and get 1/10 of the output...they're probably going to be late adopters by virtue of bad fit
- deleted 3y ago
  [deleted]
- chaostheory 3y ago
  Keyboards and a trackpads work with the Vision Pro
  - FirmwareBurner 3y ago
    But if you're gonna be tied to a keyboard and mouse on a desk then why do you need a AVP and not a highdpi monitor and spend the rest on a vacation to Hawaii/Ibiza? The whole point of Vision Pro like devices is you wear your computer on your face always with you and the only peripherals you need are your eyes and hands also always on you.
    - martin_ 3y ago
      in part, it may serve as a substitute to highdpi monitor while you’re in Hawaii
      - wkat4242 3y ago
        It's not highdpi. It's only 4K per eye and that's over the entire field of vision. Highdpi would be 4K on a 24" screen a foot or two away. The vision pro spreads those pixels out over a much bigger angle. It's got enough pixels to equal a few 1080p monitors though.
empath-nirvana 3y ago
There's probably just some UI elements people are going to have to avoid -- small, closely spaced text seems like it's likely to be a problem..
- crooked-v 3y ago
  The bare minimum usable font size feels to be about 20-24pt for me (judging off a virtual screen placed about the same distance I'd hold a laptop), so I think the only time people are even going to run into it is if they're doing something like zooming in/out an entire game UI at once.
meindnoch 3y ago
The problem is simply that their downsampling is not gamma-corrected, so overall brightness is not preserved. http://www.ericbrasseur.org/gamma.html http://www.ericbrasseur.org/gamma.html
- 01HNNWZ0MV43FF 3y ago
  > If you want to give it a try, scale it down 50% using your best software. Happy to report that in the last few years since 2007, GIMP did fix their default colorspace, I can scale the Dalai Lama down even with the Linear filter and it doesn't turn grey.
- spookie 3y ago
  Also the reason why linear gradients aren't done properly in CSS, and W3C is aware of this. https://erikmcclure.com/blog/everyone-does-srgb-wrong-because/ https://erikmcclure.com/blog/everyone-does-srgb-wrong-becaus...
  - TheCoreh 3y ago
    You can now opt in to the proper interpolation with the `in srgb-linear` option: linear-gradient(in srgb-linear, ...) Works in Safari/Chrome but not yet on Firefox
- hedgehog 3y ago
  There's some pretty glaring aliasing and moire issues. Gamma correction might matter once they try to come up with better shaders but that's not the main issue. Rendering fine detail like a spreadsheet without major artifacts is an interesting problem, burden is mostly on Apple to figure out a solution for app devs. Higher resolution screens won't really help that much.
- rpastuszak 3y ago
  I put together some notes and research on the subject for anyone interested: https://untested.sonnet.io/Natural+Gradients+in+CSS https://untested.sonnet.io/Natural+Gradients+in+CSS Not sure if it’s still up to date, so please check out the list of resources mentioned in the footer too.
miiiiiike 3y ago
And eating popcorn breaks the pinch detection.
operator-name 3y ago
Honestly, this is a really cool demonstration of how their foveated rendering works.
MPSimmons 3y ago
This isn't breaking, it's just testing the bounds. Still interesting, but not groundbreaking.
mayoff 3y ago
Watch the second half of WWDC 2023 session “Explore rendering for spatial computing” to see what Apple wants to tell developers about foveated rendering. Ivan never actual uses the term “foveated rendering” though. https://developer.apple.com/wwdc23/10095 https://developer.apple.com/wwdc23/10095 There is also an article, “Rendering at Different Rasterization Rates”: https://developer.apple.com/documentation/metal/render_passes/rendering_at_different_rasterization_rates/ https://developer.apple.com/documentation/metal/render_passe...
losvedir 3y ago
A follow-up comment by the author indicates much of this is a bug having to do with Excel specifically, since a lossless PNG screenshot doesn't exhibit this behavior.
- pixelesque 3y ago
  The use-case of viewing images is possibly (likely?) special-cased to be mip-mapped for correctly filtering / downsampling them, whereas generic applications which draw arbitrary content to the frame buffer likely don't have this?
solardev 3y ago
Is this really some unsolvable problem? Seems like a cool opportunity for UX innovation. That shimmer looks kinda like a moire effect to me, and probably isn't an issue at higher zoom levels? I wonder if there an opportunity here to explore other ways of navigating a spreadsheet, like maybe panning and zooming the UI like a map, almost. The longer you stare at an area, the more it zooms in so you can easily see just a few cells at a time. You can still have a minimap nearby to see the big overview, but your main focus would be on just a few cells.
- kevinmalone 3y ago
  This is an important point. I’ve found that 45x45 pixel buttons are as low as I’m comfortable putting in my Vision Pro apps, with roughly 10px of padding. The eye tracking is good, but there are certainly limits of precision. A spreadsheet has a LOT going on, and each cell has at least two clickable areas. The bottom right drag area is probably impossible to target right now without accidentally going into the next cell. Spreadsheets have good form for what they’re used for. The new input method (eye and pinch) is going to help people rethink methods of interaction in places where “that’s how we’ve always done it” has been a mainstay. Maybe spreadsheets stay the same, but new approaches to interacting with data come to the surface.
exitzer0 3y ago
This does not strike me as a productivity device. I doubt many of the target demographic will actually notice this issue.
notdisliked 3y ago
I noticed the same behavior in my Vision Pro the other day. The conclusion I came to was actually that I was not seeing the boundaries of foveated rendering, but a UI element. When staring at touch targets, it projects an overlay over the target with a slightly bright large circle around your eyes, fading further away from your eyes. As you move your eyes around, the circle follows them. This is particularly noticeable where a very large touch target (like a large text box) is present, you can easily see the entirety of the circle overlay, whereas with smaller targets it’s cut off on the edges of the target. It also happens to align with the foveated rendering downsampling circle because of course they both follow your eyes. I’m guessing this is what is happening here. To test this myself, I looked around on a large, lighter colored page in safari with no large touch targets, and could not see the same circular glow.
BasilPH 3y ago
For anybody else who didn't know what foveated rendering was: > Foveated imaging is a digital image processing technique in which the image resolution, or amount of detail, varies across the image according to one or more "fixation points". A fixation point indicates the highest resolution region of the image and corresponds to the center of the eye's retina, the fovea. [^0] [^0]: https://en.wikipedia.org/wiki/Foveated_imaging https://en.wikipedia.org/wiki/Foveated_imaging
MaximilianEmel 3y ago
For anyone unclear about the title: It is about the foveated rendering looking weird, not the tracking itself breaking.
tamimio 3y ago
So I assume running a terminal won’t work?

146 points by Topfi 3y ago

I cannot be the only one who reads AVP as “Alien vs. Predator”.

To be fair, this is a 1.0 version of brand new software and hardware. By the time we get to visionOS 3.0, and maybe even before that, this will be gone.

sneak 3y ago
I don’t think this will ever improve significantly (maybe a little bit with software tweaks) on AVP1 hardware. The current algorithm is not terrible; it may be possible to improve the artifacting a bit but ultimately the rendering setup exists to cope with the resolution of the displays. The state of the art in cost-effective HMDs will need to advance a bit for this to really get “fixed”, as TFA notes. It is telling that it took a very contrived test (a white grid of fine lines on black) for this to be obvious. Most things displayed by most users won’t run into this problem. I agree that this thing is not yet monitor-replacement-level yet. I mostly just want it for watching movies in bed or on planes.
MBCook 3y ago
All the “it’s not good enough” posts amaze me. As you said we’re still on 1.0. It hasn’t even been a week. I bet Apple can improve it without further hardware, let alone future versions. However I was wondering just how accurate this could get yesterday. Because of saccades the eye never really stops and just looks at one thing right? Which would mean we can never tell what someone is looking at, only the area of the eye is heavily focused around. Sort of a probability field. Just how small can we make that field and have it still feel accurate? Or does this introduce something of the “thumb“ problem on touchscreens where you have to just do your best to interpret what the user probably wanted?
- deleted 3y ago
  [deleted]
- crooked-v 3y ago
  So far with mine, it feels like fingertip size is about the smallest 'eye target' I can use for a close-up screen (think roughly the position of a held iPad) and still have 100% accuracy. I would bet that the limits of eye accuracy are the math basis of window resizing with push/pull movement, so that the accuracy rate remains the same for farther-away virtual screens.
- rozab 3y ago
  I believe eye tracking systems have been used for weapons targeting in F-35s for about 10 years. They wouldn't be using it for that usecase if it wasn't pretty accurate
  - zarzavat 3y ago
    If you’re in a fighter jet and there’s an enemy aircraft you don’t need much encouragement to look at it, whereas with computers people look away from the UI element they are interacting with all the time, for example I’m typing on the iOS keyboard now but looking at the text. Also the number of objects in a fighter jet is going to be rather small so the radius for selection around each object can be large.
  - MBCook 3y ago
    True. But an F-35 cockpit has more space for cameras/etc and a slightly higher price. I also suspect selecting from very close together (in your field of view) enemies isn’t a too common. On the other hand on a site like HN there are a ton of very small targets right next to eachother.
_ugfj 3y ago
Let's hope we never do. Of course, there's very little hope but we all would be better off if this product didn't exist. Let me quote Paris Marx's excellent summary from his Apple Vision Pro preview https://disconnect.blog/apples-vision-pro-headset-deserves/ https://disconnect.blog/apples-vision-pro-headset-deserves/ > Tech companies want us isolated and constantly staring at screens because it drives profit and of course > The Vision Pro lets users further separate themselves from their surroundings and even create “digital personas” for video calls so other people are talking to a digital, abstracted version of themselves instead of seeing the real person. Even The Verge acknowledges: > using the Vision Pro is such a lonely experience, regardless of the weird ghost eyes on the front. Of course it's Apple so it'll catch on and what in 1969 was predicted to happen in 4545 will happen, well, quite a few years earlier : In the year 4545 ... : Nobody's gonna look at you And of course https://img.ifunny.co/images/3461f2f61d0ac8385d0c1985ac415547f6a1ad99783efc91a20dbe94d4158315_1.jpg https://img.ifunny.co/images/3461f2f61d0ac8385d0c1985ac41554...
- CatWChainsaw 3y ago
  We can always hope that Vision Pro turns into Glasshole 2.0.

threeseed 3y ago

This is simply how foveated rendering works. Nothing is broken here. It's an optimisation technique that is needed now because we don't have the resources to handle the full resolution. But this will change over the coming years. For now maybe just increase the font size on the spreadsheet.

crooked-v 3y ago
The "broken" part is, outside of the visible difference in the foveation, the "fizzing" effect from whatever's going on with the antialiasing being weird with the zoomed-out grid.
- kevingadd 3y ago
  I think these are typically called moire patterns, and you will see them almost any time a high frequency signal is being rendered at a lower frequency. Anti-aliasing, box filters etc are common workarounds for them but you can't always make them completely go away.

Havoc 3y ago

Not sure it matters at this stage? One day excel gang will no doubt be using this type of VR setup. It makes for a very poor early adopter crowd though - Excel users need keyboards not current finger gesture setup. They're very reliant on muscle memory, shortcuts and predictable low latency interactions with literally a single piece of software. Unless you want to pay your accountant twice as much and get 1/10 of the output...they're probably going to be late adopters by virtue of bad fit

deleted 3y ago
[deleted]
chaostheory 3y ago
Keyboards and a trackpads work with the Vision Pro
- FirmwareBurner 3y ago
  But if you're gonna be tied to a keyboard and mouse on a desk then why do you need a AVP and not a highdpi monitor and spend the rest on a vacation to Hawaii/Ibiza? The whole point of Vision Pro like devices is you wear your computer on your face always with you and the only peripherals you need are your eyes and hands also always on you.
  - martin_ 3y ago
    in part, it may serve as a substitute to highdpi monitor while you’re in Hawaii
    - wkat4242 3y ago
      It's not highdpi. It's only 4K per eye and that's over the entire field of vision. Highdpi would be 4K on a 24" screen a foot or two away. The vision pro spreads those pixels out over a much bigger angle. It's got enough pixels to equal a few 1080p monitors though.

empath-nirvana 3y ago

There's probably just some UI elements people are going to have to avoid -- small, closely spaced text seems like it's likely to be a problem..

crooked-v 3y ago
The bare minimum usable font size feels to be about 20-24pt for me (judging off a virtual screen placed about the same distance I'd hold a laptop), so I think the only time people are even going to run into it is if they're doing something like zooming in/out an entire game UI at once.

meindnoch 3y ago

The problem is simply that their downsampling is not gamma-corrected, so overall brightness is not preserved. http://www.ericbrasseur.org/gamma.html http://www.ericbrasseur.org/gamma.html

01HNNWZ0MV43FF 3y ago
> If you want to give it a try, scale it down 50% using your best software. Happy to report that in the last few years since 2007, GIMP did fix their default colorspace, I can scale the Dalai Lama down even with the Linear filter and it doesn't turn grey.
spookie 3y ago
Also the reason why linear gradients aren't done properly in CSS, and W3C is aware of this. https://erikmcclure.com/blog/everyone-does-srgb-wrong-because/ https://erikmcclure.com/blog/everyone-does-srgb-wrong-becaus...
- TheCoreh 3y ago
  You can now opt in to the proper interpolation with the `in srgb-linear` option: linear-gradient(in srgb-linear, ...) Works in Safari/Chrome but not yet on Firefox
hedgehog 3y ago
There's some pretty glaring aliasing and moire issues. Gamma correction might matter once they try to come up with better shaders but that's not the main issue. Rendering fine detail like a spreadsheet without major artifacts is an interesting problem, burden is mostly on Apple to figure out a solution for app devs. Higher resolution screens won't really help that much.
rpastuszak 3y ago
I put together some notes and research on the subject for anyone interested: https://untested.sonnet.io/Natural+Gradients+in+CSS https://untested.sonnet.io/Natural+Gradients+in+CSS Not sure if it’s still up to date, so please check out the list of resources mentioned in the footer too.

miiiiiike 3y ago

And eating popcorn breaks the pinch detection.

operator-name 3y ago

Honestly, this is a really cool demonstration of how their foveated rendering works.

MPSimmons 3y ago

This isn't breaking, it's just testing the bounds. Still interesting, but not groundbreaking.

mayoff 3y ago

Watch the second half of WWDC 2023 session “Explore rendering for spatial computing” to see what Apple wants to tell developers about foveated rendering. Ivan never actual uses the term “foveated rendering” though. https://developer.apple.com/wwdc23/10095 https://developer.apple.com/wwdc23/10095 There is also an article, “Rendering at Different Rasterization Rates”: https://developer.apple.com/documentation/metal/render_passes/rendering_at_different_rasterization_rates/ https://developer.apple.com/documentation/metal/render_passe...

losvedir 3y ago

A follow-up comment by the author indicates much of this is a bug having to do with Excel specifically, since a lossless PNG screenshot doesn't exhibit this behavior.

pixelesque 3y ago
The use-case of viewing images is possibly (likely?) special-cased to be mip-mapped for correctly filtering / downsampling them, whereas generic applications which draw arbitrary content to the frame buffer likely don't have this?

solardev 3y ago

Is this really some unsolvable problem? Seems like a cool opportunity for UX innovation. That shimmer looks kinda like a moire effect to me, and probably isn't an issue at higher zoom levels? I wonder if there an opportunity here to explore other ways of navigating a spreadsheet, like maybe panning and zooming the UI like a map, almost. The longer you stare at an area, the more it zooms in so you can easily see just a few cells at a time. You can still have a minimap nearby to see the big overview, but your main focus would be on just a few cells.

kevinmalone 3y ago
This is an important point. I’ve found that 45x45 pixel buttons are as low as I’m comfortable putting in my Vision Pro apps, with roughly 10px of padding. The eye tracking is good, but there are certainly limits of precision. A spreadsheet has a LOT going on, and each cell has at least two clickable areas. The bottom right drag area is probably impossible to target right now without accidentally going into the next cell. Spreadsheets have good form for what they’re used for. The new input method (eye and pinch) is going to help people rethink methods of interaction in places where “that’s how we’ve always done it” has been a mainstay. Maybe spreadsheets stay the same, but new approaches to interacting with data come to the surface.

exitzer0 3y ago

This does not strike me as a productivity device. I doubt many of the target demographic will actually notice this issue.

notdisliked 3y ago

I noticed the same behavior in my Vision Pro the other day. The conclusion I came to was actually that I was not seeing the boundaries of foveated rendering, but a UI element. When staring at touch targets, it projects an overlay over the target with a slightly bright large circle around your eyes, fading further away from your eyes. As you move your eyes around, the circle follows them. This is particularly noticeable where a very large touch target (like a large text box) is present, you can easily see the entirety of the circle overlay, whereas with smaller targets it’s cut off on the edges of the target. It also happens to align with the foveated rendering downsampling circle because of course they both follow your eyes. I’m guessing this is what is happening here. To test this myself, I looked around on a large, lighter colored page in safari with no large touch targets, and could not see the same circular glow.

BasilPH 3y ago

For anybody else who didn't know what foveated rendering was: > Foveated imaging is a digital image processing technique in which the image resolution, or amount of detail, varies across the image according to one or more "fixation points". A fixation point indicates the highest resolution region of the image and corresponds to the center of the eye's retina, the fovea. [^0] [^0]: https://en.wikipedia.org/wiki/Foveated_imaging https://en.wikipedia.org/wiki/Foveated_imaging

MaximilianEmel 3y ago

For anyone unclear about the title: It is about the foveated rendering looking weird, not the tracking itself breaking.

tamimio 3y ago

So I assume running a terminal won’t work?