3 ms·

The curious thing about the past, present and future of AR... if there was low hanging fruit (in terms of irresistible applications) that people must absolutely

The curious thing about the past, present and future of AR... if there was low hanging fruit (in terms of irresistible applications) that people must absolutely have, love at first "sight" experiences, wouldn't it have happened already? with whatever crude software and hardware? Not as a gimmick but as persistent interest and investment[0]. People got busy with computers when computers were the size of houses. They carried mobile phones when phones were the size of suitcases. Is the proto-AR revolution happening somewhere out there without anybody noticing? Its possible that there is no low hanging AR fruit. That somebody has to do the difficult job of climbing up the AR tree so to speak (refine the technology until it feels like magic). It just seems that this would be an exception in how things played out so far in the "digital transformation" journey. Its the nature of the human brain to fill-in the gaps and overlook the rough edges when it really has an incentive to do so. [0] investment in the sense of personal time by users, creators, business people etc to really learn and use the technology to scratch whatever itch they found it is scratching...

greggman3 5y ago
there is no low hanging fruit. For AR to really work so many things need to happen. For one, it arguably requires even more CPU/GPU than VR because unlike current phones and current VR you really want ALL the apps to run concurrently in AR. You want your social media app to show the names and or interests of the people around you. You want your mapping app to show you directions. You want your restaurant app to point out which restaurants have seats available or a low waiting time and price / cuisine / style. You want your virtual pet to run around your feet and body, you want to see each person's virtual fashion, you want your virtual monitors showing your older 2d apps running, you want your virtual cooking instructions and while sometimes you'll want to turn down one app over another, usually you'll want many running at the same time highlighting different things. You can even argue you want the various apps to share 3d spacial data (sans privacy issues) so your virtual pet can interact with your friend's virtual pet and the virtual furniture they have placed in their house via some furniture app. one app at a time is just not good enough for the general population to get iPhone level interested in AR. Getting full view glasses with fast enough depth sensing so things can be correctly occluded and run all apps simalutainiously and have the batteries last all day seems many years away
- nexuist 5y ago
  I'm not a graphics expert, but the reason why VR is so intensive is because it has to render the same frame twice from two different perspectives (one for each eye). This is compounded by the fact that low resolution / frame-rate VR hurts to look at, and so there is a minimum resolution requirement on top of a minimum frame rate requirement on top of a double render requirement. There's a lot of compute power necessary to build a smooth VR experience. AR doesn't have to have those same requirements because you're not going to get nauseous looking at a buggy label flickering in the distance. You don't have to spend compute cycles to trick your brain into believing what you're seeing is real, because it is actually real (minus the overlayed 3D models, of course). Now, sure, you will have to spend compute cycles on depth sensing and mapping your immediate area. But that's something the OS should/will do, not every app simultaneously. If you think about what an AR operating system would be responsible for doing, mapping your surroundings and providing that data via API is probably one of the first features it would have. It's no different from Windows or macOS communicating with your monitor so that your applications can draw on it. Similarly, every app likely won't be responsible for drawing its UX onto the user's vision - it will probably submit some graphic or model to the OS, which will then anchor that model "onto" reality and handle the user moving their head around it. Much like how every Windows app is not responsible for resizing or moving its window, that's the window manager's job to do. In AR we would probably have a reality manager or something. What I will agree with you on is that we probably need desktop-level rendering power to solve AR, not mobile-level. However, with the release of M1, it does seem like we're pretty much there already, and I would not bet against Apple's chip team failing to make a smaller M1 that fits into AR glasses.
  - pshc 5y ago
    I’m also not an expert but I believe that AR is that much more difficult because any latency between the projected image and the real world will show up as displacement when you move or turn your head. Whereas in VR, because the world is obscured, the brain will accept your artificial world even if there’s ~20ms latency.
  - greggman3 5y ago
    > every app likely won't be responsible for drawing its UX onto the user's vision - it will probably submit some graphic or model to the OS, which will then anchor that model "onto" reality and handle the user moving their head around it. Much like how every Windows app is not responsible for resizing or moving its window, that's the window manager's job to do. In AR we would probably have a reality manager or something. I don't think this is true. The problem with an app just giving 3D data to the OS is then no apps can compete on quality. There's a reason Cyberpunk 2077 (or whatever the top graphics game is) looks different than Wii Bowling. Even with 2D windows each app does its own rendering using the GPU and provides the results as a rectangle of pixels that the app computed. Some apps look amazing because they used 100% of the GPU power. Some look so-so or maybe they have a great aesthetic (Animal Crossing?) but they aren't spending power on rendering. Now, add that across apps. Your virtual pet app wants to render photo realistic fur where the fur appears to bend in relation to things in the real world, your virtual fashion app wants to show you're friends outrageous auto-reactive clothing, but your navigation app just wants to draw some solid more mostly solid colored lines. The virtual pet app will hog all of the GPU just to draw its fur bogging the entire OS down. It won't be a simple as it was for desktop PCs because you can't let any app slow down the system (which is what happens on PCs) since you need the system to always run at 60 or 90 or 120fps. At best you could maybe make a preemptable GPU (not sure any current GPUs are preemptable), or use 2 GPUs, one for the OS, and one for apps. I think Apple or someone will ship an AR device without solving these issues. I think it will fail just like the Apple Newton failed. Too early. Running AR glasses with an iOS like OS that runs one app at a time is just not a useful paradigm for AR and solving the problem of letting 5 to 20 apps all run at the same time, each using as much GPU as possible for their specific idea of quality, AND have that all run fast enough to keep up with latency issues to actually have the virtual stuff not swim over the real stuff just has a long ways to go. Maybe an M17 but not a M1
astrange 5y ago
I believe it is happening and it's called Pokemon Go. There should be many more industrial applications than consumer ones though, eg imagine what construction workers could do with it. The main problem is that it's so personal and low latency, but it also takes so much processing power. You need to spend a lot of time on performance and power work before anyone will use it without their phone melting.

greggman3 5y ago

there is no low hanging fruit. For AR to really work so many things need to happen. For one, it arguably requires even more CPU/GPU than VR because unlike current phones and current VR you really want ALL the apps to run concurrently in AR. You want your social media app to show the names and or interests of the people around you. You want your mapping app to show you directions. You want your restaurant app to point out which restaurants have seats available or a low waiting time and price / cuisine / style. You want your virtual pet to run around your feet and body, you want to see each person's virtual fashion, you want your virtual monitors showing your older 2d apps running, you want your virtual cooking instructions and while sometimes you'll want to turn down one app over another, usually you'll want many running at the same time highlighting different things. You can even argue you want the various apps to share 3d spacial data (sans privacy issues) so your virtual pet can interact with your friend's virtual pet and the virtual furniture they have placed in their house via some furniture app. one app at a time is just not good enough for the general population to get iPhone level interested in AR. Getting full view glasses with fast enough depth sensing so things can be correctly occluded and run all apps simalutainiously and have the batteries last all day seems many years away

nexuist 5y ago
I'm not a graphics expert, but the reason why VR is so intensive is because it has to render the same frame twice from two different perspectives (one for each eye). This is compounded by the fact that low resolution / frame-rate VR hurts to look at, and so there is a minimum resolution requirement on top of a minimum frame rate requirement on top of a double render requirement. There's a lot of compute power necessary to build a smooth VR experience. AR doesn't have to have those same requirements because you're not going to get nauseous looking at a buggy label flickering in the distance. You don't have to spend compute cycles to trick your brain into believing what you're seeing is real, because it is actually real (minus the overlayed 3D models, of course). Now, sure, you will have to spend compute cycles on depth sensing and mapping your immediate area. But that's something the OS should/will do, not every app simultaneously. If you think about what an AR operating system would be responsible for doing, mapping your surroundings and providing that data via API is probably one of the first features it would have. It's no different from Windows or macOS communicating with your monitor so that your applications can draw on it. Similarly, every app likely won't be responsible for drawing its UX onto the user's vision - it will probably submit some graphic or model to the OS, which will then anchor that model "onto" reality and handle the user moving their head around it. Much like how every Windows app is not responsible for resizing or moving its window, that's the window manager's job to do. In AR we would probably have a reality manager or something. What I will agree with you on is that we probably need desktop-level rendering power to solve AR, not mobile-level. However, with the release of M1, it does seem like we're pretty much there already, and I would not bet against Apple's chip team failing to make a smaller M1 that fits into AR glasses.
- pshc 5y ago
  I’m also not an expert but I believe that AR is that much more difficult because any latency between the projected image and the real world will show up as displacement when you move or turn your head. Whereas in VR, because the world is obscured, the brain will accept your artificial world even if there’s ~20ms latency.
- greggman3 5y ago
  > every app likely won't be responsible for drawing its UX onto the user's vision - it will probably submit some graphic or model to the OS, which will then anchor that model "onto" reality and handle the user moving their head around it. Much like how every Windows app is not responsible for resizing or moving its window, that's the window manager's job to do. In AR we would probably have a reality manager or something. I don't think this is true. The problem with an app just giving 3D data to the OS is then no apps can compete on quality. There's a reason Cyberpunk 2077 (or whatever the top graphics game is) looks different than Wii Bowling. Even with 2D windows each app does its own rendering using the GPU and provides the results as a rectangle of pixels that the app computed. Some apps look amazing because they used 100% of the GPU power. Some look so-so or maybe they have a great aesthetic (Animal Crossing?) but they aren't spending power on rendering. Now, add that across apps. Your virtual pet app wants to render photo realistic fur where the fur appears to bend in relation to things in the real world, your virtual fashion app wants to show you're friends outrageous auto-reactive clothing, but your navigation app just wants to draw some solid more mostly solid colored lines. The virtual pet app will hog all of the GPU just to draw its fur bogging the entire OS down. It won't be a simple as it was for desktop PCs because you can't let any app slow down the system (which is what happens on PCs) since you need the system to always run at 60 or 90 or 120fps. At best you could maybe make a preemptable GPU (not sure any current GPUs are preemptable), or use 2 GPUs, one for the OS, and one for apps. I think Apple or someone will ship an AR device without solving these issues. I think it will fail just like the Apple Newton failed. Too early. Running AR glasses with an iOS like OS that runs one app at a time is just not a useful paradigm for AR and solving the problem of letting 5 to 20 apps all run at the same time, each using as much GPU as possible for their specific idea of quality, AND have that all run fast enough to keep up with latency issues to actually have the virtual stuff not swim over the real stuff just has a long ways to go. Maybe an M17 but not a M1

astrange 5y ago

I believe it is happening and it's called Pokemon Go. There should be many more industrial applications than consumer ones though, eg imagine what construction workers could do with it. The main problem is that it's so personal and low latency, but it also takes so much processing power. You need to spend a lot of time on performance and power work before anyone will use it without their phone melting.