3 ms·

> Latency's a big one, especially if you're doing head-tracking or augmented reality. From a coworker's Twitter feed: https://twitter.com/#!/rygorous/statuses

> Latency's a big one, especially if you're doing head-tracking or augmented reality. From a coworker's Twitter feed: https://twitter.com/#!/rygorous/statuses/168173568889327616 https://twitter.com/#!/rygorous/statuses/168173568889327616 Numbers: E2E latency of key press->VGA signal or keyboard LED toggle: both ~23.3ms. Measured with 100MHz analog oscilloscope+300fps cam. For that experiment he used a little custom circuit for input (button) and output (LED) that was wired to the serial port of a computer running Windows. In addition to the output LED, the button was also directly wired to another LED so as to get precise measurements of the delay. If you were to have full end-to-end control of the system and you chose to make it a top property, you could obviously get much lower latency. But to even get those numbers I believe he had to use a special driver that granted his test program privileged access so he could do direct port IO with INB and OUTB rather than delegate the work to the kernel.

epaik 15y ago
I think the big latency issue here isn't so much with analog input delay of the user as it is the speed of video processing algorithms. Currently researched augmented reality solutions look great, but they're plagued by delay that make usability an issue. As processing power becomes cheaper and faster, it makes sense that we'll eventually be able to achieve almost-realtime augmented reality. At such a time in the near future, head mounted displays will definitely become popular.
- psykotic 15y ago
  My point was more that the baseline latency on a modern computer and operating system, without really doing anything on the application side, is already very significant--it took simplifying assumptions and a good deal of hacking to get to that figure of ~23.3 ms. So, yes, latency is a big issue with practical augmented reality, on several levels of the hardware/software stack. Computer vision algorithms that rely on temporal coherence can easily introduce many frames of latency, so even if the per-frame processing time is manageable, the latency may not be.
  - epaik 15y ago
    I agree. The problem of latency definitely exists on more than the software level. From my anecdotal experience however, the majority of current latency in augmented reality does seem to lie in the vision processing. Interestingly, previous research[1] indicates the maximal latency for a mobile augmented reality device to be fully immersive is 5-10 ms. While I don't think this target is reachable with modern technology, as you've obviously made clear, I feel we'll eventually get to a point where it's Good Enough for practical use. I can't help but feel excited about what kinds of applications such technology will yield in the future. [1]Pasman, W., Schaaf, A. van der, Lagendijk, R. L., & Jansen, F. W. (1999). Low latency rendering for mobile augmented reality. Proceedings of the ASCI'99 (Heijen, the Netherlands, June 15-17), 372-376.
    - groby_b 15y ago
      I wonder if "Good Enough" really is. One big problem with latency is that it may well lead to motion sickness, since your perception of motion from the visual system and your inner ear is out of sync. Disclaimer: It's been a _long_ time since I worked with heads-up/AR displays. Let's make it 15 years. (I feel ancient now ;)
    - ekianjo 15y ago
      Actually if you do want to "feel" the latency at all in most situations, you'd rather be at 1ms than 10 ms. 10ms still carries a huge lag when you are moving you head around in several directions. 1 ms brings it way closer to being real. We need improvements of a number of magnitudes to reach high resolution, processed graphics and display at 1ms rate to work. We may have some working prototypes within one decade, with commercial applications 10-15 years from now. It could go faster than that, but not very likely.
      - bane 15y ago
        Related to your point. Microsoft Research and some prototype displays. http://www.youtube.com/watch?feature=player_embedded&v=vOvQCPLkPt4 http://www.youtube.com/watch?feature=player_embedded&v=v...
- cma 15y ago
  The main delay issues are in head movement, hand tracking etc. is much more forgivable. Thankfully head tracking can be augmented with low-latency accelerometer data.

epaik 15y ago

I think the big latency issue here isn't so much with analog input delay of the user as it is the speed of video processing algorithms. Currently researched augmented reality solutions look great, but they're plagued by delay that make usability an issue. As processing power becomes cheaper and faster, it makes sense that we'll eventually be able to achieve almost-realtime augmented reality. At such a time in the near future, head mounted displays will definitely become popular.

psykotic 15y ago
My point was more that the baseline latency on a modern computer and operating system, without really doing anything on the application side, is already very significant--it took simplifying assumptions and a good deal of hacking to get to that figure of ~23.3 ms. So, yes, latency is a big issue with practical augmented reality, on several levels of the hardware/software stack. Computer vision algorithms that rely on temporal coherence can easily introduce many frames of latency, so even if the per-frame processing time is manageable, the latency may not be.
- epaik 15y ago
  I agree. The problem of latency definitely exists on more than the software level. From my anecdotal experience however, the majority of current latency in augmented reality does seem to lie in the vision processing. Interestingly, previous research[1] indicates the maximal latency for a mobile augmented reality device to be fully immersive is 5-10 ms. While I don't think this target is reachable with modern technology, as you've obviously made clear, I feel we'll eventually get to a point where it's Good Enough for practical use. I can't help but feel excited about what kinds of applications such technology will yield in the future. [1]Pasman, W., Schaaf, A. van der, Lagendijk, R. L., & Jansen, F. W. (1999). Low latency rendering for mobile augmented reality. Proceedings of the ASCI'99 (Heijen, the Netherlands, June 15-17), 372-376.
  - groby_b 15y ago
    I wonder if "Good Enough" really is. One big problem with latency is that it may well lead to motion sickness, since your perception of motion from the visual system and your inner ear is out of sync. Disclaimer: It's been a _long_ time since I worked with heads-up/AR displays. Let's make it 15 years. (I feel ancient now ;)
  - ekianjo 15y ago
    Actually if you do want to "feel" the latency at all in most situations, you'd rather be at 1ms than 10 ms. 10ms still carries a huge lag when you are moving you head around in several directions. 1 ms brings it way closer to being real. We need improvements of a number of magnitudes to reach high resolution, processed graphics and display at 1ms rate to work. We may have some working prototypes within one decade, with commercial applications 10-15 years from now. It could go faster than that, but not very likely.
    - bane 15y ago
      Related to your point. Microsoft Research and some prototype displays. http://www.youtube.com/watch?feature=player_embedded&v=vOvQCPLkPt4 http://www.youtube.com/watch?feature=player_embedded&v=v...
cma 15y ago
The main delay issues are in head movement, hand tracking etc. is much more forgivable. Thankfully head tracking can be augmented with low-latency accelerometer data.