6 ms·

I'm not sure if there is a good short primer on Retinal Displays or not, but I don't think this is it. As the article itself outlines, it's been decades of res

I'm not sure if there is a good short primer on Retinal Displays or not, but I don't think this is it. As the article itself outlines, it's been decades of research and we haven't really been able to make any practical progress yet. There are a lot of reasons for this that are as much social as they are technical. At my previous company a few years ago we took a crack at a concept [1], but never got it off the ground because prototype costs were in the millions and there are only a few places that we could source the kind of hardware you need to build these things. Last I checked in 2017, Oculus/Facebook was making the most progress on this but even then were having trouble miniaturizing things. I'm really curious where they are with it. [1] https://pdfaiw.uspto.gov/.aiw?docid=20180131926&SectionNum=1&IDKey=363814FA3947&HomeUrl=http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1%2526Sect2=HITOFF%2526d=PG01%2526p=1%2526u=%25252Fnetahtml%25252FPTO%25252Fsrchnum.html%2526r=1%2526f=G%2526l=50%2526s1=%25252220180131926%252522.PGNR.%2526OS=DN/20180131926%2526RS=DN/20180131926 https://pdfaiw.uspto.gov/.aiw?docid=20180131926&SectionNum=1...

qznc 7y ago
Sounds like we wait for some technologies to reduce the prototype cost by two orders of magnitude. What makes it so costly?
- mojomark 7y ago
  > What makes it so costly? I've seen this question a few times, so I'll try to address... Yes, you can take a bi-axial rastor scanning MEMS mirror (similar to what Microvision uses in their Pico projector, or those supplied by Mirrorcle), shine it into your pupil (at the appropriate) distance and BAM!, you have a point source VRD. However, I would heed the warning on Microvisions Pico projector as those ARE NOT eye safe lasers. It is nit difficult, however, to get your hands on eyesafe laser sources suitable for VRD. However, with this simple device your FOV will be limited by the angular range of your MEMS. Your resolution will be limited by the pulse width of your synchronized laser. Most importantly though is that the exit pupil (eyebox) is very small such that when you move your eyeball a few degrees, the image will start to disappear. So, you need redundant point sources to expand the FOV and also expand the eyebox to permit eye rotation. These point sources must be spaced in a matrix of less than 2mmx2mm (the typical minimum dilation of the pupil in bright ambient settings). That's a lot of point sources, so if you try to just use a bunch of MEMS mirrors, the driving electronics will generally occlude your natural view of the environment. That's not a problem for video see-through displays, but for optical see-through displays that's a deal breaker. So, you're back to trying to disassociate the light beam point sources from the scanning-driver electronics. The typical strategy is to use some sort of waveguide - to transport the light beam (for a given portion to the image) through a transparent material (e.g. glass), then get that beam to outcouple from the waveguide at the precise location, then have that light beam continue to raster scan. Now you need to do that for, a few hundred light beams (per eye!) to fully fill the natural human FOV with a composite image greater than 35 pixels per degree (for text readability) All of the above items are achievable, but it requires precision optics and alignment to get everything working in harmony. Add the component of affordability, and you've got a pretty big engineering challenge on your hand. I'm sure someone can do it. This is just one viable path to truly transformative AR displays, but I think it's the most promising.
mhh__ 7y ago
I wouldn't be surprised if Valve have at least thought about going down this route. They seem to be really enthusiastic (perhaps terrifyingly) about Brain Computer Interfaces, being able to project directly onto the Eye in a VR headset might be a good stepping stone for them.
- krenzo 7y ago
  In an interview, Jeri Ellsworth stated that when she was at Valve around 2012, they were exploring all types of near-eye displays for use in their eventual VR headset, and one of the displays they tested was a laser retinal projection display. It was specifically mentioned because she pointed out the safety concern they had with projecting a laser into the eye.
- polytely 7y ago
  Last GDC there was a cool BCI talk from Valve[0], if I had money I would bet that the next VR headset from valve would contain some brain activity sensors. And in this interview with IGN, Gabe Newel talks a bit about where they stand with BCI [1]. Gabe: "reading and writing to someones motor-cortex is much more of a tractable problem than making someone feel cold" The idea that different parts of the brain have different levels of interface-usability was really mind-blowing to me. I guess it's easy to fall into the trap of seeing the brain as one thing, while it's doing a lot of different things which you can interact with in different ways. [0]: https://www.youtube.com/watch?v=Qhj3C1H5JWo https://www.youtube.com/watch?v=Qhj3C1H5JWo [1]: https://youtu.be/I0zXkwLs_lo?t=647 https://youtu.be/I0zXkwLs_lo?t=647
  - taneq 7y ago
    > Gabe: "reading and writing to someones motor-cortex is much more of a tractable problem than making someone feel cold" This is probably very true, but that doesn't mean that accessing the motor cortex is easy. In prosthetics, the two workable approaches currently are an implanted BCI, or using bits of muscle as 'amplifiers' to turn the motor nerve signals into something that can be read via an external electrode. We're a LONG way from being able to parse an EEG into physical sensations in any practical sense.
raywu 7y ago
Hi Andrew, I saw your VB FB AR article from 2017 and checked out Pair3D. Can you comment on Bosch's approach vs Intel Vaunt (now defunct)?
- AndrewKemendo 7y ago
  I liked both and they seemed like good first starts from what I saw. Never got to personally try them so my knowledge is pretty limited.
krenzo 7y ago
What exactly cost so much? You can build a retinal display using a $200 off-the-shelf laser projector if you turn down the power and add less than $100 in lenses. I've done it, and I've even managed to get it up to 8K resolution with some more time and money thrown at it. I've since formed a company to try to commercialize it: http://www.alphalux.io http://www.alphalux.io
- AndrewKemendo 7y ago
  Without knowing more about your solution I can't comment on that specifically. I do know that the mems, fiber coupling to the mems and the waveguides necessary for wide enough FOV and variable focal length are very much bespoke right now and thus costly.
- kempbellt 7y ago
  Not sure what you are attempting to commercialize, but your website doesn't explain at all what you have made. Unless I'm supposed to guess it from your background picture of 3d rendered glasses with some sensors in them... From most studies I've read on the subject, there are a couple big issues. Firstly, physics is hard. It's very difficult to get a useful amount of digital information presented that closely to your eye, where it is in focus, and comfortable to look at. Scaling up resolution beyond a low-res screen is even more difficult. Secondly, tech just isn't there yet. The closest thing I've seen in the market is Google Glass, and that was a huge flop. It also looked dorky as hell, while providing very little real value. Sure, you could read a text, if you squinted and focused your attention up and to the right, but at that point, pulling out your phone is just as easy. It it also makes you look like a strange android while taking your attention away from the real world. For a product like this to work in the market it has to meet a lot of requirements. Resolution, invisibility (as in, it doesn't feel like you're wearing a clunky awkward device on your face), battery life, safety (you're shining light into your eyes), and provide real, useful, functionality. Useful functionality, to make it worth caring about a device like this, is good augmented reality integration. And I mean very good. If you put the device on your face and the mapping of the real world stutters for a second, you're going to hate it and never use it - and no one will buy it. Also, no, being able to project a little cartoon monster on the surface of a table is not "useful" AR functionality...

qznc 7y ago

Sounds like we wait for some technologies to reduce the prototype cost by two orders of magnitude. What makes it so costly?

mojomark 7y ago
> What makes it so costly? I've seen this question a few times, so I'll try to address... Yes, you can take a bi-axial rastor scanning MEMS mirror (similar to what Microvision uses in their Pico projector, or those supplied by Mirrorcle), shine it into your pupil (at the appropriate) distance and BAM!, you have a point source VRD. However, I would heed the warning on Microvisions Pico projector as those ARE NOT eye safe lasers. It is nit difficult, however, to get your hands on eyesafe laser sources suitable for VRD. However, with this simple device your FOV will be limited by the angular range of your MEMS. Your resolution will be limited by the pulse width of your synchronized laser. Most importantly though is that the exit pupil (eyebox) is very small such that when you move your eyeball a few degrees, the image will start to disappear. So, you need redundant point sources to expand the FOV and also expand the eyebox to permit eye rotation. These point sources must be spaced in a matrix of less than 2mmx2mm (the typical minimum dilation of the pupil in bright ambient settings). That's a lot of point sources, so if you try to just use a bunch of MEMS mirrors, the driving electronics will generally occlude your natural view of the environment. That's not a problem for video see-through displays, but for optical see-through displays that's a deal breaker. So, you're back to trying to disassociate the light beam point sources from the scanning-driver electronics. The typical strategy is to use some sort of waveguide - to transport the light beam (for a given portion to the image) through a transparent material (e.g. glass), then get that beam to outcouple from the waveguide at the precise location, then have that light beam continue to raster scan. Now you need to do that for, a few hundred light beams (per eye!) to fully fill the natural human FOV with a composite image greater than 35 pixels per degree (for text readability) All of the above items are achievable, but it requires precision optics and alignment to get everything working in harmony. Add the component of affordability, and you've got a pretty big engineering challenge on your hand. I'm sure someone can do it. This is just one viable path to truly transformative AR displays, but I think it's the most promising.

mhh__ 7y ago

I wouldn't be surprised if Valve have at least thought about going down this route. They seem to be really enthusiastic (perhaps terrifyingly) about Brain Computer Interfaces, being able to project directly onto the Eye in a VR headset might be a good stepping stone for them.

krenzo 7y ago
In an interview, Jeri Ellsworth stated that when she was at Valve around 2012, they were exploring all types of near-eye displays for use in their eventual VR headset, and one of the displays they tested was a laser retinal projection display. It was specifically mentioned because she pointed out the safety concern they had with projecting a laser into the eye.
polytely 7y ago
Last GDC there was a cool BCI talk from Valve[0], if I had money I would bet that the next VR headset from valve would contain some brain activity sensors. And in this interview with IGN, Gabe Newel talks a bit about where they stand with BCI [1]. Gabe: "reading and writing to someones motor-cortex is much more of a tractable problem than making someone feel cold" The idea that different parts of the brain have different levels of interface-usability was really mind-blowing to me. I guess it's easy to fall into the trap of seeing the brain as one thing, while it's doing a lot of different things which you can interact with in different ways. [0]: https://www.youtube.com/watch?v=Qhj3C1H5JWo https://www.youtube.com/watch?v=Qhj3C1H5JWo [1]: https://youtu.be/I0zXkwLs_lo?t=647 https://youtu.be/I0zXkwLs_lo?t=647
- taneq 7y ago
  > Gabe: "reading and writing to someones motor-cortex is much more of a tractable problem than making someone feel cold" This is probably very true, but that doesn't mean that accessing the motor cortex is easy. In prosthetics, the two workable approaches currently are an implanted BCI, or using bits of muscle as 'amplifiers' to turn the motor nerve signals into something that can be read via an external electrode. We're a LONG way from being able to parse an EEG into physical sensations in any practical sense.

raywu 7y ago

Hi Andrew, I saw your VB FB AR article from 2017 and checked out Pair3D. Can you comment on Bosch's approach vs Intel Vaunt (now defunct)?

AndrewKemendo 7y ago
I liked both and they seemed like good first starts from what I saw. Never got to personally try them so my knowledge is pretty limited.

krenzo 7y ago

What exactly cost so much? You can build a retinal display using a $200 off-the-shelf laser projector if you turn down the power and add less than $100 in lenses. I've done it, and I've even managed to get it up to 8K resolution with some more time and money thrown at it. I've since formed a company to try to commercialize it: http://www.alphalux.io http://www.alphalux.io

AndrewKemendo 7y ago
Without knowing more about your solution I can't comment on that specifically. I do know that the mems, fiber coupling to the mems and the waveguides necessary for wide enough FOV and variable focal length are very much bespoke right now and thus costly.
kempbellt 7y ago
Not sure what you are attempting to commercialize, but your website doesn't explain at all what you have made. Unless I'm supposed to guess it from your background picture of 3d rendered glasses with some sensors in them... From most studies I've read on the subject, there are a couple big issues. Firstly, physics is hard. It's very difficult to get a useful amount of digital information presented that closely to your eye, where it is in focus, and comfortable to look at. Scaling up resolution beyond a low-res screen is even more difficult. Secondly, tech just isn't there yet. The closest thing I've seen in the market is Google Glass, and that was a huge flop. It also looked dorky as hell, while providing very little real value. Sure, you could read a text, if you squinted and focused your attention up and to the right, but at that point, pulling out your phone is just as easy. It it also makes you look like a strange android while taking your attention away from the real world. For a product like this to work in the market it has to meet a lot of requirements. Resolution, invisibility (as in, it doesn't feel like you're wearing a clunky awkward device on your face), battery life, safety (you're shining light into your eyes), and provide real, useful, functionality. Useful functionality, to make it worth caring about a device like this, is good augmented reality integration. And I mean very good. If you put the device on your face and the mapping of the real world stutters for a second, you're going to hate it and never use it - and no one will buy it. Also, no, being able to project a little cartoon monster on the surface of a table is not "useful" AR functionality...