7 ms·

Uploadvr demystified this stuff a couple years ago: http://uploadvr.com/magic-leap-how-it-works/ http://uploadvr.com/magic-leap-how-it-works/ Magic leap is a

Uploadvr demystified this stuff a couple years ago: http://uploadvr.com/magic-leap-how-it-works/ http://uploadvr.com/magic-leap-how-it-works/ Magic leap is a pair of glasses you wear on your face that displays AR imagery through a clear glass lens. There's a cable leading from these glasses to your pocket where you have a tablet-size device doing all the heavy lifting, most likely using mobile SoC's. Pretty sure you can't run a desktop CPU and GPU on a pocketable device. You don't get 1080gtx performance or anything remotely close to that with a setup like this. Its mobile-level graphics. Oculus's recent showing of its mobile SoC-based Santa Cruz prototype shows fairly poor graphics compared to PC-based VR like the Rift or Vive. I imagine the Magic Leap has less pixels to push considering it isn't doing any backgrounds, but still, those very impressive marketing shots of high resolution dragons flying around and interacting with you might not really be possible without a lot of compromises on graphic quality, fov, framerate, etc. Worse, no one wants to talk about transparency, especially in well-lit rooms. The few hands-on Hololens reports we've gotten make this out to be a big issue, as well as fov, and fundamentally the Magic Leap and what hololens is using could be the same or very similar technologies. My gut feeling is that no one has yet to handle the transparency issue (and this is why MS can show off the hololens with confidence) and because of that it makes sense for Magic Leap to keep its product secretive. The tech press was not too kind to Hololens prototypes due to transparecy issues and Magic Leap doesn't need that kind of negative press as it continues to fund-raise and develop its product.

neuronexmachina 10y ago
How many years will it be until Pascal-like performance is available in a tablet form-factor?
- drzaiusapelord 10y ago
  Maybe never. Pascal is 16nm process and the mid-range gaming card, the 1070, uses up to 150 watts. A "powerful" mobile chipset uses a tiny fraction of that even at its highest settings. 100 watts for the CPU and another 150watts for the GPU is a staggering 250watts of continuous power no pocket-able lithium-ion battery can handle. If we want current gen graphics we're more than likely looking at a PC backpack solution. More than likely we'll probably be looking at some low watt solution, like the 35W AMD chipset in the Solus Q and accept a lower level of graphic quality, but still well above mobile standards. This may involve wearing a battery belt or battery hip-pouch for extended use. For lower run times the battery can be part of the headstrap with no need for anything to be pocketable.
  - kbenson 10y ago
    Part of me says just distribute the battery all around, thighs, legs, torso and you'll at least have good power. Another part of me says this is ludicrous, and is akin to strapping thermite and a bunch of loose strike anywhere matches into pouches all around your body.
    - jpalomaki 10y ago
      For many professional applications it is not hard to imagine people wearing some kind of vest or belt related to this.
- barrkel 10y ago
  Only by some combination of streaming and prediction; doing the rendering in the cloud, and using clever technology to hide latency.
- KaoruAoiShiho 10y ago
  Which Pascal? Assuming the smallest desktop Pascal, GP107 is at 768 CUDA cores. Tegra X2 Pascal has around 256 CUDA cores. They seem to be clocked similarly. The Tegra also has a lot lower memory bandwidth and stuff like that but we'll put that aside for now. We'll need the Tegra to x3 in performance. Maybe in 6? years we can get a GP107 performance in tablet form factor. The GTX 1080 has 2560 CUDA cores. That's another 3x from GP107. So maybe 12 years for ~1080 performance in tablets.
kimburgess 10y ago
One important differentiating factor between this and other (consumer) HMD's is the scanning display. The Oculus et al need to render a rectangular matrix of evenly spaced pixels. If you want a larger field of view or higher resolution this comes with an exponential increase in number of pixels. With a high frequency scanning display and eye tracking you are not locked to a static spatial resolution (see https://en.m.wikipedia.org/wiki/Foveated_imaging https://en.m.wikipedia.org/wiki/Foveated_imaging). There's a whole bunch of other problems when it comes to a render pipeline for something links this, but at least on the physical front it enables super high perceived resolution with less overall points of light to render.
- pfranz 10y ago
  Foveated rendering only became relevant when Nvidia recently started supporting Multi-Resolution Shading. As new techniques and tools like this get better support we can be a lot more efficient with lower powered hardware. It was interesting to hear Carmack last year talk about re-introducing interlacing as a way to reduce latency. Although, I'm just reinforcing what you're saying =)
  - xorxornop 10y ago
    Huh, interlacing is a neat idea to reduce latency, but I'm guessing you'd want a form of intra-frame temporal anti-aliasing, otherwise it might look pretty fugly. Whether or not this would neutralise any latency advantage due to the computational overhead of doing this, I'm not sure...
    - pfranz 10y ago
      Found where Carmack talked about it[1]. For VR, the consensus I've seen seem to push for multi-sample AA > full-screen AA > temporal AA (I feel like I've even seen no AA is better than temporal AA). I'm not quite sure if it's performance, architecture (they really prefer forward renderers instead of deferred renderers for low latency), or aesthetics. In the little bit I've done, when just playing with knobs temporal AA looks better to me. Without AA specular highlights are way too distracting and pop too much. This reference[2] talks about the extra velocity buffer needed for temporal AA and the fact it tends to over-blur which can fuzz out fine details (low resolution is a touchy subject in VR). There's a lot of moving pieces when making decisions. In talking about interlacing, he was ideally talking about not having to create a whole image buffer before sending data to the hardware--just rendering parts (well, scanline) of the image that moved. [1] https://youtu.be/gn8m5d74fk8?t=10m54s https://youtu.be/gn8m5d74fk8?t=10m54s [2] https://www.youtube.com/watch?v=Xk3WUk5T2TQ&feature=youtu.be&t=2525 https://www.youtube.com/watch?v=Xk3WUk5T2TQ&feature=youtu.be...
digi_owl 10y ago
Reminds me that a company a while back tried to pitch such a system on the cheap. It was basically a bulky visor that you put your smartphone into. inside the visor was a angled piece of glass that would make your phone screen appear to be hovering in the air in front of you.
- ebalit 10y ago
  An AR Cardboard would be awesome. I searched some weeks ago if someone worked on this without luck. Do you remember the name of this company?
  - digi_owl 10y ago
    http://www.caputer.com/ http://www.caputer.com/
bobsil1 10y ago
- Unlike VR, they don't have to render backgrounds - FOV is apparently their big advantage, along with focus zones - They could do hard occlusion with an LCD mask. Or they could punt til v2, and soft-occlude with blur using their stacked diffraction plates.
- mojomark 10y ago
  > - They could do hard occlusion with an LCD mask. Or they could punt til v2, and soft-occlude with blur using their stacked diffraction plates. Unfortunately, you cannot easily achieve "hard occlusion" with an LCD mask. Pinlight display is a project that did explore that idea: http://pinlights.info http://pinlights.info LCDs can attenuate individual rays of light. However, the images we see of real objects in our environment are generated from light in the form of planar wave fronts. So, if you have a near eye LCD micro display and you turn on a single LCD pixel in an attempt to block light emanating from a certain point on a real object, most of the light emanating from that point will simply go around the LCD pixel, enter the pupil and for a point on you retina. The total brightness of the image will dim a little dependant on the size of the LCD pixel, but not by much for a single microdisplay pixel (i.e. a few micron pitch). The Pinlight papers do talk about ways around that issue, using LCD diffraction pattern masks, but I still think the residual diffraction and reduced overall lens transparency associated with the LCD micro display makes the approach an even bigger challenge.
  - bobsil1 10y ago
    Depends on size of occluded area? If I block the sun with my thumb, some light does go around, but it's still a decently effective mask.
    - mojomark 10y ago
      If you hold out your thumb between your eye and the sun, such that your thumb appears to just block the edges of the sun, then you will be effectively blocking the entire set of overlapping wavefronts emanating from the sun (that would have formed multiple "pixels" forming the sun image on your retina), preventing their arrival at your cornea and ultimately entering your pupil. Notice that your thumb is about about 1.5ft from your eye. Could you emulate that with a transparent LCD display? Yes, but that display would need to be large and about 1.5ft from your eye. Not really practical for a near eye (head mounted display). If you try to accomplish this occlusion directly with an LCD microdiplay (such as those found in many projectors) in a near eye application, at best you end up with a blackish smudge with very fuzzy edges. To see this in action, look at Figure 11.D in the following paper: http://www.cs.unc.edu/~maimone/media/pinlights_siggraph_2014.pdf http://www.cs.unc.edu/~maimone/media/pinlights_siggraph_2014...
mojomark 10y ago
> There's a cable leading from these glasses to your pocket where you have a tablet-size device doing all the heavy lifting, most likely using mobile SoC's. Pretty sure you can't run a desktop CPU and GPU on a pocketable device. > You don't get 1080gtx performance or anything remotely close to that with a setup like this. Its mobile-level graphics. A.) the pocket device is the light source array that feed the scanning fibers. A processor drives each light source linearly. Neither the light source nor the driving processor are likely to be head mounted for quite some time, but I don't see that being an issue given the dramatic improvement over existing interfaces (i.e. I'd rather have a pair of glasses with a tether than a helmet any day of the week); B.) In light field VRDs, resolution is not limited by active matrix array addressing as it is to a degree in LCD displays of mobile devices. Instead, resolution primarily depends on how quickly you can resonate the tip of a scanning fiber and concurrently how quickly you can pulse that fiber's light source in a synchronized manner (each pulse matching the bit color/intensity of the intended pixel); C.) Multiple fibers form a composite image; D.) Maximum resolution is only needed in the foveal region (i.e. a few deg about the center of user's eye gaze at any given time), so eye tracking can be utilized to minimize processing. The remainder of displayed content is low resolution, albeit still critical, peripheral imagery. My team & I have been batting around with a slightly different approach to the light field VRD, which we're thinking about simply releasing as a FOSH project.
- yunolisten 10y ago
  > we're thinking about simply releasing as a FOSH project. More details would be fantastic
  - mojomark 10y ago
    Well, our light field display concept is fairly different from ML's in the manner in which we scan the light into the eye. We propose using very narrow collimated wavefronts instead of broad/potentially curved wavefronts. The system can still be used to support accomodation, potentially to a greater degree than is possible using layered DOE's (I don't know for sure) which we do not propose to use. The resultant display has an inherently smaller exit pupil, but we'll layout a method to actively expand the exit pupil, enough to support typical maximum eye rotation range and minimum eye pupil dilation (~2mm) without vignetting the virtual image with natural eye movements. The benefit of this method is virtually unlimitted FOV. Certainly still in development, but we think releasing what we've come up with thus far into the wild as FOSH and seeing what the crowd can bring to the design is a fun idea. I'm currently working on a simple "Hello World" write up that will explain the device components, some basic plans for prototyping done to date and the development hurdles that still need to be overcome. I'm planning to use this to introduce the project. It's definitely not an easy project, but I don't see any component that can't be fabricated by a crafty homebrew engineer.
SolarNet 10y ago
There is a bigger problem, as well. Tracking, the glasses will need to use a lot of the GPU (and CPU) processing power for simply tracking the environment. So they don't have backgrounds, that doesn't account for much anyway, like maybe 10% of GPU power goes to that. Tracking an environment takes like an entire card at the moment.
- schnelle 10y ago
  Microsoft created an ASIC they call an HPU (holographic processing unit) that does this work for the GPU. It takes data from the depth sensors and sends the resulting point data straight to the CPU. One of the researchers posted a 30ish min video on the hardware inside the Hololens on YouTube. It's this HPU / sensors that Microsoft is licensing to the hardware manufacturers for the upcoming VR and AR headsets. We'll know more December 8th, and after their hardware expo later this year.

neuronexmachina 10y ago

How many years will it be until Pascal-like performance is available in a tablet form-factor?

drzaiusapelord 10y ago
Maybe never. Pascal is 16nm process and the mid-range gaming card, the 1070, uses up to 150 watts. A "powerful" mobile chipset uses a tiny fraction of that even at its highest settings. 100 watts for the CPU and another 150watts for the GPU is a staggering 250watts of continuous power no pocket-able lithium-ion battery can handle. If we want current gen graphics we're more than likely looking at a PC backpack solution. More than likely we'll probably be looking at some low watt solution, like the 35W AMD chipset in the Solus Q and accept a lower level of graphic quality, but still well above mobile standards. This may involve wearing a battery belt or battery hip-pouch for extended use. For lower run times the battery can be part of the headstrap with no need for anything to be pocketable.
- kbenson 10y ago
  Part of me says just distribute the battery all around, thighs, legs, torso and you'll at least have good power. Another part of me says this is ludicrous, and is akin to strapping thermite and a bunch of loose strike anywhere matches into pouches all around your body.
  - jpalomaki 10y ago
    For many professional applications it is not hard to imagine people wearing some kind of vest or belt related to this.
barrkel 10y ago
Only by some combination of streaming and prediction; doing the rendering in the cloud, and using clever technology to hide latency.
KaoruAoiShiho 10y ago
Which Pascal? Assuming the smallest desktop Pascal, GP107 is at 768 CUDA cores. Tegra X2 Pascal has around 256 CUDA cores. They seem to be clocked similarly. The Tegra also has a lot lower memory bandwidth and stuff like that but we'll put that aside for now. We'll need the Tegra to x3 in performance. Maybe in 6? years we can get a GP107 performance in tablet form factor. The GTX 1080 has 2560 CUDA cores. That's another 3x from GP107. So maybe 12 years for ~1080 performance in tablets.

kimburgess 10y ago

One important differentiating factor between this and other (consumer) HMD's is the scanning display. The Oculus et al need to render a rectangular matrix of evenly spaced pixels. If you want a larger field of view or higher resolution this comes with an exponential increase in number of pixels. With a high frequency scanning display and eye tracking you are not locked to a static spatial resolution (see https://en.m.wikipedia.org/wiki/Foveated_imaging https://en.m.wikipedia.org/wiki/Foveated_imaging). There's a whole bunch of other problems when it comes to a render pipeline for something links this, but at least on the physical front it enables super high perceived resolution with less overall points of light to render.

pfranz 10y ago
Foveated rendering only became relevant when Nvidia recently started supporting Multi-Resolution Shading. As new techniques and tools like this get better support we can be a lot more efficient with lower powered hardware. It was interesting to hear Carmack last year talk about re-introducing interlacing as a way to reduce latency. Although, I'm just reinforcing what you're saying =)
- xorxornop 10y ago
  Huh, interlacing is a neat idea to reduce latency, but I'm guessing you'd want a form of intra-frame temporal anti-aliasing, otherwise it might look pretty fugly. Whether or not this would neutralise any latency advantage due to the computational overhead of doing this, I'm not sure...
  - pfranz 10y ago
    Found where Carmack talked about it[1]. For VR, the consensus I've seen seem to push for multi-sample AA > full-screen AA > temporal AA (I feel like I've even seen no AA is better than temporal AA). I'm not quite sure if it's performance, architecture (they really prefer forward renderers instead of deferred renderers for low latency), or aesthetics. In the little bit I've done, when just playing with knobs temporal AA looks better to me. Without AA specular highlights are way too distracting and pop too much. This reference[2] talks about the extra velocity buffer needed for temporal AA and the fact it tends to over-blur which can fuzz out fine details (low resolution is a touchy subject in VR). There's a lot of moving pieces when making decisions. In talking about interlacing, he was ideally talking about not having to create a whole image buffer before sending data to the hardware--just rendering parts (well, scanline) of the image that moved. [1] https://youtu.be/gn8m5d74fk8?t=10m54s https://youtu.be/gn8m5d74fk8?t=10m54s [2] https://www.youtube.com/watch?v=Xk3WUk5T2TQ&feature=youtu.be&t=2525 https://www.youtube.com/watch?v=Xk3WUk5T2TQ&feature=youtu.be...

digi_owl 10y ago

Reminds me that a company a while back tried to pitch such a system on the cheap. It was basically a bulky visor that you put your smartphone into. inside the visor was a angled piece of glass that would make your phone screen appear to be hovering in the air in front of you.

ebalit 10y ago
An AR Cardboard would be awesome. I searched some weeks ago if someone worked on this without luck. Do you remember the name of this company?
- digi_owl 10y ago
  http://www.caputer.com/ http://www.caputer.com/

bobsil1 10y ago

- Unlike VR, they don't have to render backgrounds - FOV is apparently their big advantage, along with focus zones - They could do hard occlusion with an LCD mask. Or they could punt til v2, and soft-occlude with blur using their stacked diffraction plates.

mojomark 10y ago
> - They could do hard occlusion with an LCD mask. Or they could punt til v2, and soft-occlude with blur using their stacked diffraction plates. Unfortunately, you cannot easily achieve "hard occlusion" with an LCD mask. Pinlight display is a project that did explore that idea: http://pinlights.info http://pinlights.info LCDs can attenuate individual rays of light. However, the images we see of real objects in our environment are generated from light in the form of planar wave fronts. So, if you have a near eye LCD micro display and you turn on a single LCD pixel in an attempt to block light emanating from a certain point on a real object, most of the light emanating from that point will simply go around the LCD pixel, enter the pupil and for a point on you retina. The total brightness of the image will dim a little dependant on the size of the LCD pixel, but not by much for a single microdisplay pixel (i.e. a few micron pitch). The Pinlight papers do talk about ways around that issue, using LCD diffraction pattern masks, but I still think the residual diffraction and reduced overall lens transparency associated with the LCD micro display makes the approach an even bigger challenge.
- bobsil1 10y ago
  Depends on size of occluded area? If I block the sun with my thumb, some light does go around, but it's still a decently effective mask.
  - mojomark 10y ago
    If you hold out your thumb between your eye and the sun, such that your thumb appears to just block the edges of the sun, then you will be effectively blocking the entire set of overlapping wavefronts emanating from the sun (that would have formed multiple "pixels" forming the sun image on your retina), preventing their arrival at your cornea and ultimately entering your pupil. Notice that your thumb is about about 1.5ft from your eye. Could you emulate that with a transparent LCD display? Yes, but that display would need to be large and about 1.5ft from your eye. Not really practical for a near eye (head mounted display). If you try to accomplish this occlusion directly with an LCD microdiplay (such as those found in many projectors) in a near eye application, at best you end up with a blackish smudge with very fuzzy edges. To see this in action, look at Figure 11.D in the following paper: http://www.cs.unc.edu/~maimone/media/pinlights_siggraph_2014.pdf http://www.cs.unc.edu/~maimone/media/pinlights_siggraph_2014...

mojomark 10y ago

> There's a cable leading from these glasses to your pocket where you have a tablet-size device doing all the heavy lifting, most likely using mobile SoC's. Pretty sure you can't run a desktop CPU and GPU on a pocketable device. > You don't get 1080gtx performance or anything remotely close to that with a setup like this. Its mobile-level graphics. A.) the pocket device is the light source array that feed the scanning fibers. A processor drives each light source linearly. Neither the light source nor the driving processor are likely to be head mounted for quite some time, but I don't see that being an issue given the dramatic improvement over existing interfaces (i.e. I'd rather have a pair of glasses with a tether than a helmet any day of the week); B.) In light field VRDs, resolution is not limited by active matrix array addressing as it is to a degree in LCD displays of mobile devices. Instead, resolution primarily depends on how quickly you can resonate the tip of a scanning fiber and concurrently how quickly you can pulse that fiber's light source in a synchronized manner (each pulse matching the bit color/intensity of the intended pixel); C.) Multiple fibers form a composite image; D.) Maximum resolution is only needed in the foveal region (i.e. a few deg about the center of user's eye gaze at any given time), so eye tracking can be utilized to minimize processing. The remainder of displayed content is low resolution, albeit still critical, peripheral imagery. My team & I have been batting around with a slightly different approach to the light field VRD, which we're thinking about simply releasing as a FOSH project.

yunolisten 10y ago
> we're thinking about simply releasing as a FOSH project. More details would be fantastic
- mojomark 10y ago
  Well, our light field display concept is fairly different from ML's in the manner in which we scan the light into the eye. We propose using very narrow collimated wavefronts instead of broad/potentially curved wavefronts. The system can still be used to support accomodation, potentially to a greater degree than is possible using layered DOE's (I don't know for sure) which we do not propose to use. The resultant display has an inherently smaller exit pupil, but we'll layout a method to actively expand the exit pupil, enough to support typical maximum eye rotation range and minimum eye pupil dilation (~2mm) without vignetting the virtual image with natural eye movements. The benefit of this method is virtually unlimitted FOV. Certainly still in development, but we think releasing what we've come up with thus far into the wild as FOSH and seeing what the crowd can bring to the design is a fun idea. I'm currently working on a simple "Hello World" write up that will explain the device components, some basic plans for prototyping done to date and the development hurdles that still need to be overcome. I'm planning to use this to introduce the project. It's definitely not an easy project, but I don't see any component that can't be fabricated by a crafty homebrew engineer.

SolarNet 10y ago

There is a bigger problem, as well. Tracking, the glasses will need to use a lot of the GPU (and CPU) processing power for simply tracking the environment. So they don't have backgrounds, that doesn't account for much anyway, like maybe 10% of GPU power goes to that. Tracking an environment takes like an entire card at the moment.

schnelle 10y ago
Microsoft created an ASIC they call an HPU (holographic processing unit) that does this work for the GPU. It takes data from the depth sensors and sends the resulting point data straight to the CPU. One of the researchers posted a 30ish min video on the hardware inside the Hololens on YouTube. It's this HPU / sensors that Microsoft is licensing to the hardware manufacturers for the upcoming VR and AR headsets. We'll know more December 8th, and after their hardware expo later this year.