11 ms·

John Carmack: Latency Mitigation Strategies

257 points by bigdubs 14y ago

modeless 14y ago
I think head-mounted-display virtual reality is a huge missed opportunity for next-gen consoles. They could have really done it right in a way that PCs can't yet, with special hardware support for low latency and stereo rendering, and a guaranteed large audience so game studios could justify applying their full budgets to AAA VR titles. It would be a huge differentiator at a time when consoles seem to be converging to a very similar place. It could be as big as the Wii was.
- micheljansen 14y ago
  I think the monumental failure of Nintendo's Virtual Boy pretty strongly illustrated that if you can't do this perfectly, you shouldn't do it at all. Within the current constraints (including budget), making a "good enough" product is clearly not feasible. As this article points out, even a few milliseconds too much causes massive problems (worst case: severe nausea – car sickness). On top of that, it's not even clear that people want this. Full-on Virtual Reality is a bit antisocial and hyper realism isn't the only thing that makes games immersive and/or fun.
  - modeless 14y ago
    Virtual Boy pretty strongly illustrated that if you can't do this perfectly, you shouldn't do it at all. Absolutely. But the Virtual Boy was clearly before its time. The hardware just wasn't ready. Within the current constraints (including budget), making a "good enough" product is clearly not feasible. I think that's clearly not true. The response from everyone who's tried the Oculus Rift has been that real VR has finally arrived. And the Oculus Rift is one guy's prototype; the resources of a company like Microsoft or Sony could drastically improve it in several dimensions. Specifically, they could source a higher-res, higher frame rate, lighter, lower-latency OLED display, and connect it to rendering hardware with all the latency-reducing features John Carmack just described. These features aren't necessarily expensive from a hardware POV; it's just that nobody's seen the need to implement them yet.
    - neverm0re 14y ago
      > Specifically, they could source a higher-res, higher frame rate, lighter, lower-latency OLED display None of those are actually remotely as important as the fact you're still using your head as a dumb camera joystick, there's no ability to track eye movement and the mismatch between what you're seeing and your inner-ear pressure is going to make a number of people throw up.[1] These are fundamental problems that have been a part of HMDs for the last twenty years of me playing with them and the Rift addresses absolutely none of them. The Rift itself is not significantly different than its HMD contemporaries, it's actually aiming for the low-end gamer market and so far seems to be mostly pushing 'What's old is new again' without having made significant progress along the way. The most interesting part of the Rift kit is the tracker, not the actual HMD itself. [1]: In a study conducted by U.S. Army Research Institute for the Behavioral and Social Sciences in a report published May 1995 titled "Technical Report 1027 - Simulator Sickness in Virtual Environments", out of 742 pilot exposures from 11 military flight simulators, "approximately half of the pilots (334) reported post-effects of some kind: 250 (34%) reported that symptoms dissipated in less than 1 hour, 44 (6%) reported that symptoms lasted longer than 4 hours, and 28 (4%) reported that symptoms lasted longer than 6 hours. There were also 4 (1%) reported cases of spontaneously occurring flashbacks." Read up on 'Simulation Sickness' for more details.
      - modeless 14y ago
        Eye tracking would be awesome but it's not required for good VR. I fail to see how "using your head as a dumb camera joystick" is a problem; that's the whole point of VR. Motion sickness is a real problem for some people (but not all). With low latency and thoughtful game design I think it can be mitigated. The bigger problem is the social acceptability of blocking your entire field of vision for long periods of time. I won't pretend that VR doesn't have problems, but the payoff is large enough that the problems are worth tackling.
        neverm0re 14y ago
        > I fail to see how "using your head as a dumb camera joystick" is a problem; that's the whole point of VR. Perhaps you should try playing some VR games for a while. I clocked quite a few hours playing the old Virtuality SU2000 games like Dactyl Nightmare and I've idly kept up with HMDs. The chief problem is that without eye tracking, it's incredibly UNFUN to use your head for movements that your eyes could otherwise have done for you. Mind you when the HMDs were much heavier back then it sucked a lot more, but it's still pretty shitty not being able to glance aside. Nope, gotta move your entire head for absolutely everything related to what you're currently seeing, if you move your head for any reason you can't maintain focus on objects naturally, etc. > Motion sickness is a real problem for some people (but not all). Probably not something you should underestimate. See Nintendo's 3DS launch and about-face on their stance on pushing 3D once they found a small but significant percentage of their users could not actually see the 3D effect. This lead to policy that the 3D effect could not be used for anything related to actual gameplay mechanics, reducing it entirely to an optional gimmick. Simulation sickness affects even more people than the 3D issue. It's a real problem if you want to go mainstream. Just to clarify my position, I'm not against the Rift nor do I have anything against HMDs. I simply see the Rift as a step along the way to whatever device truly popularizes the tech. I don't think we're there yet.
- deleted 14y ago
  [deleted]
- untog 14y ago
  I think VR is fundamentally broken, because you're going to try to react to things in a way that doesn't work. With a head-mounted display, the game can react as you move your head. Great. But then you'll instinctively try to walk towards something and it won't work (or you'll hit the wall). Same with picking things up, etc. It seems like it's deep in the uncanny valley of experience.
  - WesleyJohnson 14y ago
    I think it's just a matter of time before the other pieces to the VR puzzle catch up. Omni-directional treadmills are already in existence and are a basic step towards allowing you to walk in a VR environment. Developments in prosthetics with 'feeling' could help pave way towards shoes, gloves or other garments that could provide you with the sensory experiences of walking on different surfaces or picking up in-game items.
    - untog 14y ago
      But at that point you've gone from a games console being something that takes up a small amount of space underneath your TV to something that takes up an entire room. From requiring you to pick up a controller to putting on an entire outfit. I'm just not sure the average consumer is in any way ready/willing for that.
      - purplelobster 14y ago
        I think the first successful VR you'll see will be closer to an arcade machine than a home console.
        stcredzero 14y ago
        > I think the first successful VR you'll see will be closer to an arcade machine than a home console. I was just thinking about this. My development rig has 2 27" screens. If I added a 3rd one, this would cover most of my field of view horizontally. What if someone sold a console rig packaged with 3 screen and a mount? (Work with IKEA to produce a compatible cheap desk.) This kind of thing is already done and is awesome for vehicular games. (Racing, tank, aircraft...) A 3 screen rig also could still strike a kid as "awesome," so would draw money from the wallets of parents. Or, for arcade-type settings, how about a seat with a projected 360 dome, HOTAS controls, and motion simulation?
- ChuckMcM 14y ago
  I don't think they were a missed opportunity. In order for them to 'miss' they had to be 'possible' which they haven't been for a long time. Its only lately where various things have come together to get close. I have followed the 'head mounted display' technology closely since its early inception in the mid-90's as a means of providing military pilots with situational awareness. The first problem with head mounted displays was resolution. Both in color space and in pixel space. Early displays were monochrome (either green or red based on LED display drivers) and had a roughly 256 x 256 dot pitch. It was ok for targeting reticules and basic instrumentation display (like an artificial horizon). Early work also wanted transparency (look through) displays because there wasn't a way to display what you were looking at in enough fidelity to do both the visual field and the indicator field. Later a company called Colorado Micro Displays (CMD) came to market and started offering color, but 320 x 200 was the best resolution they could do. Higher resolution displays cost tens of thousands of dollars and were essentially hand crafted out of unobtainum. Direct retinal illumination displays attempted to get past strict display resolution issues (a friend of mine worked on one of the x-y positioning device for doing this, it had 500 angstrom repeatability! And was, as expected, insanely expensive) It really hasn't been until recently when DLP type systems at sufficient resolution to eliminate the x-y type beam director have come to market. Combined with better LED phosphors to bring the cost of an RGB display with better than VGA (640 x 480) resolution into something practical. The incessant push to higher resolution phone screens has created the capability to make opaque but high resolution screens which can enable something like the Oculus Rift type displays. That requirement stems from how far away from the eyes you have to put the screen and thus the torque moment that is applied to your head. There was a great system at NASA which used 10" displays but the 'head mounted' part was more like sticking your head into some weird steadycam kind of device. Finally there is the challenge of both high fidelity and high frequency head orientation technologies. Prior to about 2005 the best you could do was a laser gyro for motion and an accelerometer for inertial reference. Doing that at the necessary frequency (typically 1000 updates per second) didn't become cost effective until about 2009. Now however, many of the technologies have finally matured to the point where OR can be done in small quantities for perhaps $2K/unit. A 'game console' is struggles if it costs more then $300. So as a 'controller' option, and even as the 'whole console' option, these sorts of glasses are still about 3 - 5 years from hitting a price point that makes them the 'killer' peripheral. And of course because they aren't here yet, they can't really have 'missed' :-)
- mtgx 14y ago
  Well Oculus Rift should support everything from PC's to consoles. But I agree. Stuff like Oculus Rift is the true next-gen computing. The PS4 and next Xbox leave me pretty cold, especially when I already have a laptop that can already play all the latest games.
  - nitrogen 14y ago
    I think the PS4's insane memory bandwidth (8GB of memory on a 170GB/sec GDDR5 bus) makes it a very interesting contender against PCs (dual-channel DDR3 maxes out at 34GB/s).
    - roc 14y ago
      Given the PS4 is using an off-the-shelf x86 processor, how long do you think it'll be until other OEMs are pitching laptops with these same CPUs that have GDDR5 controllers for main memory? How long until Intel matches? How long until basically every CPU supports it and every manufacturer has one or more such systems?
      - nitrogen 14y ago
        They may be using an off-the-shelf x86 core, but their actual CPU die may have all kinds of customizations on it (like the addition of a GDDR5 memory controller). AIUI GDDR5 memory is much more expensive than DDR3. So it may happen eventually, but I don't think it will be any time soon, and believe me, I really want PCs to be better than consoles in every way.
        roc 14y ago
        It just seems to me that whatever they're doing with AMD, AMD is going to be able to farm out to other parties. Given that both MS and Sony are using AMD cores paired with GDDR5 memory controllers, it seems almost certain that there's nothing approaching an exclusivity contract to prevent AMD from now selling that bundle to any maker of gaming PCs or SteamBox builds or whatever else. And the current/historical cost differential between DDR and GDDR5 certainly seemed to be a considerable problem back when these specs were first rumored. But the following things suggest that's not likely to be an issue preventing it from hitting other gaming PCs: 1. Sony managed to spec 8GB of it into a box they simply can't afford to subsidize greatly at the outset. [1] 2. Sony and MS both need to know the cost of any expensive components will drop massively over the next couple years. 3. PC gamers readily and regularly accept higher prices for their machines than console makers could ever stomach. So I don't see it becoming common. Truly very few non-gamers would care about the performance delta. But it seems like a thing that will be on offer from multiple vendors, for feasible prices, before long. (Particularly given that AMD could really use any edge against Intel.) [1] People wrote off a lot of the rumors on that basis alone. If Sony is confident that it can not only afford GDDR5 for system memory, but that it can afford so much, there has to be a massive price cut incoming.
efdee 14y ago
I feel stupid.
- MattDL 14y ago
  Honestly I think the vast majority of people (even on HN) feel fairly stupid when you read something like that. It's an utter blessing that there are people such as Carmack in the industry.
  - BystanderX 14y ago
    - who are willing/eager to share their research and thoughts with the general public. I imagine he had to push it a little to get Zenimax to allow it to be publicized, since he slipped that credit in there.
- breckinloggins 14y ago
  The view update part was the most confusing, but I tried to understand it this way: Imagine you had a magical game engine that rendered the entire world perfectly accurately for every point in space and direction a viewer could possibly be looking at. All you had to do was say: RenderGameFrameForEveryPossiblePoint(); ... // Who knows how much time viewerPosition = QuicklyGetViewerPosition(); TellTheGPUToShowWorldAccordingTo(viewerPosition); Well then, you could postpone calling those last two functions until the absolute last minute. This way you have very little or no movement of the viewer's head between when you read their position and when you show the view for that position. But naturally, RenderGameFrameForEveryPossiblePoint() is slightly out of bounds of current technology. A lot of what Carmack was discussing, as I understood it, was simulating this effect as closely as possible. The way to do that, it seems, is: StartRenderingGameFrameAccordingTo(lastViewerPosition); ... // Stuff viewerPosition = QuicklyGetViewerPosition(); viewMatrix = ComputeViewMatrixDelta(lastViewerPosition, viewerPosition); FinalizeGPUFrameRenderWithNewViewMatrix(viewMatrix); That final bit is just a perspective transformation of a bunch of rendering that was already computed and given to the GPU. But if the viewer moves too quickly, you can easily move somewhere in the world that wasn't actually rendered, or your perspective could shift such that and object that was once occluded is now visible, or vice versa. It seems a lot of the complexity is there. The last thing he talked about, time warping, seems to be a similar thing only it's scanline by scanline. So in effect you're saying "hey, video card and display, I know you're going to force me to draw a whole frame at once, so I'm going to give you a frame where each scanline gets rendered a little bit into the future according to where the player is moving." The effect on a monitor would probably look like a forward shear, but on an HMD (if done correctly), it would correct for the natural shear caused by having to "freeze frame" the viewer's perspective for one entire frame instead of just a scanline. Some of this may be woefully incorrect, but it was how I explained it to myself. Please correct anything that's wrong or overly simplified.
pdog 14y ago
With the proliferation of smartphones and projects like Google Glass and Oculus Rift taking off, I really think we'll start to see custom chipsets to improve the performance of augmented and virtual reality systems (battery life, lower latency, and more processing power).
networked 14y ago
A related article from Valve's Michael Abrash that may be a better introduction to the matter: http://blogs.valvesoftware.com/abrash/latency-the-sine-qua-non-of-ar-and-vr/ http://blogs.valvesoftware.com/abrash/latency-the-sine-qua-n... https://news.ycombinator.com/item?id=4985100 https://news.ycombinator.com/item?id=4985100
breckinloggins 14y ago
It sounds like we would benefit from GPU and display manufacturers providing a set of standard low-level control primitives so that people like Carmack who know what they are doing can really play around with the entire pipeline without having to worry about all the things the cards and the displays are doing behind their backs. For example, a GPU could have a set of standard settings with full buffering and all the other things that "help them win the framerate wars", but a developer should be able to turn all of that off when needed. It's the same with displays. LCD manufacturers could, I suppose, allow a modern day "CRT Mode 13h" where you just have scan lines mapped to memory buffers and whatever shows up in those buffers gets turned into a pixel as quickly as possible. Are there technical challenges preventing this from happening, or is it mainly inertia and lack of need from the current market?
- niggler 14y ago
  "a set of standard low-level control primitives ... Are there technical challenges preventing this from happening, or is it mainly inertia and lack of need from the current market?" It's easier to iterate faster without having to deal with a standardization committee. See OpenGL history for some context.
- Scene_Cast2 14y ago
  Regarding your last point: it's getting better. Modern TVs have "game mode", where latency is minimized. The latest LCD panels use embedded Display Port, eDP. The latest eDP standard introduces a frame buffer on the LCD itself, where you can just write deltas and tell it to "swap". I think this is quite similar to what you're proposing, in concept.
  - breckinloggins 14y ago
    I found this [1], which seems to be a good introduction to the new features and also why eDP is poised to replace LVDS (which I, as a non hardware guy, had to deal with in a previous embedded device project and hope to never deal with again). [1] http://embedded-computing.com/articles/embedded-greater-display-efficiency/ http://embedded-computing.com/articles/embedded-greater-disp...
stcredzero 14y ago
> Updating the imagery in a head mounted display (HMD) based on a head tracking sensor is a subtly different challenge than most human / computer interactions. Doesn't the military already have this solved for head mounted displays for attack helicopters and 4++ generation fighter jets? Heck, they have augmented reality displays for that matter. EDIT: Many of these problems could be solved by putting an entire purpose-built gaming rig in the headset. There are nicely capable chips for mobile devices with lots of power in the GPU. Design such a system from the ground-up for low latency. Accelerometer and head-tracking inputs would be low-level interrupts, for example. I wonder if this is how the military contractors solved this?
- knodi 14y ago
  Yes, they have with very powerful hardware thats not very consumer wallet friendly.
  - stcredzero 14y ago
    > very powerful hardware thats not very consumer wallet friendly. Yes, but the mobile landscape may well have changed this.
- niggler 14y ago
  "Doesn't the military already have this solved for head mounted displays for attack helicopters and 4++ generation fighter jets? " Far from it. There was a recall a few years ago over glitches in a fighter jet HMD. It's still an active area of research and no less than 5 different systems employed by the US military divisions
stesch 14y ago
Anyone any experience with a Wrap 1200VR from Vuzix?
Suan 14y ago
Misread as "Legacy Mitigation Strategies" - that would have been a good read as well.
dchichkov 14y ago
> Conventional computer interfaces are generally not as latency demanding as virtual reality, but sensitive users can tell the difference in mouse response down to the same 20 milliseconds or so, making it worthwhile to apply these techniques even in applications without a VR focus. I wish people would start treating text editors that they are developing this way. As hard real time systems. Don't care much about virtual reality, but I'm sick of text (and code) editors with unpredictable latency of basic operations [like typing a character, making find call, etc].
- scott_s 14y ago
  Minor quibble: you mean soft real time systems: http://en.wikipedia.org/wiki/Real-time_computing#Criteria_for_real-time_computing http://en.wikipedia.org/wiki/Real-time_computing#Criteria_fo...
- monochromatic 14y ago
  What editor are you using, and how old is your computer? I haven't had problems like this in years.
- kragen 14y ago
  Darius Bacon and I have spent some time trying to figure out a buffer data structure that can scale to large files while guaranteeing predictable latency for basic operations like typing a character. (What's "making find call"?) I think you mean "soft real time", though. A hard real-time system is one which is unusable if it fails to meet a deadline even once; a jet engine control system, say, where failing to meet a deadline could result in engine parts penetrating the fuselage. A "soft real time" system is one where failing to meet a deadline is a failure, but tolerable if infrequent; say, once every million keystrokes, or every 7 days of video play time.
- stcredzero 14y ago
  > I wish people would start treating text editors that they are developing this way. IDEs. Most of the things people do in modern IDEs were done in Smalltalk (albeit, with uglier interfaces and some manual steps) over a decade back. One of the differences, though, is that these things were mostly highly responsive, because they were bloat-free. (Can't speak for IBM VisualAge. Also, one could rightly complain that they were feature-poor in their stock configurations.) This is in stark contrast to other IDEs and app server frameworks, which would take forever to start up, or forever to restart, or forever to complete an operation. One example, Extract Method, was always instantaneous. It's still lugubrious even in modern IDEs like XCode4.5. The same rules that apply to Web Apps also apply to programmer's tools. Delays are yuck. Responsiveness is yum! I'm also a musician, and from what I've seen playing around in Audacity, 5 milliseconds is definitely inaudible, even to some of the most talented, golden-eared musicians I know. I suspect that 20 milliseconds is going to start to bug some people, though they won't be able to say why. Also, awareness of subtle modulations of very small variations in rhythm and how they affect the feel of music seems to be an indicator of intelligence and aesthetic awareness. So often, the difference between blah and great involves things that are subtle and hard to put your finger on.
ansible 14y ago
You could cut latency further if you were able to measure the muscle contractions in your neck area. Another option is smart eye tracking too. Generally if you are going to move your head in a particular direction, you'll likely start moving your eyes first.
sultezdukes 14y ago
is there anyboy smarter than carmacck..maybe sweeneye
cousin_it 14y ago
It's amazing how much time Carmack spends working around the "added cleverness" of systems created by people, rather than the inherent difficulties of the problem. These days it also seems true about programming in general.

257 points by bigdubs 14y ago

I think head-mounted-display virtual reality is a huge missed opportunity for next-gen consoles. They could have really done it right in a way that PCs can't yet, with special hardware support for low latency and stereo rendering, and a guaranteed large audience so game studios could justify applying their full budgets to AAA VR titles. It would be a huge differentiator at a time when consoles seem to be converging to a very similar place. It could be as big as the Wii was.

micheljansen 14y ago
I think the monumental failure of Nintendo's Virtual Boy pretty strongly illustrated that if you can't do this perfectly, you shouldn't do it at all. Within the current constraints (including budget), making a "good enough" product is clearly not feasible. As this article points out, even a few milliseconds too much causes massive problems (worst case: severe nausea – car sickness). On top of that, it's not even clear that people want this. Full-on Virtual Reality is a bit antisocial and hyper realism isn't the only thing that makes games immersive and/or fun.
- modeless 14y ago
  Virtual Boy pretty strongly illustrated that if you can't do this perfectly, you shouldn't do it at all. Absolutely. But the Virtual Boy was clearly before its time. The hardware just wasn't ready. Within the current constraints (including budget), making a "good enough" product is clearly not feasible. I think that's clearly not true. The response from everyone who's tried the Oculus Rift has been that real VR has finally arrived. And the Oculus Rift is one guy's prototype; the resources of a company like Microsoft or Sony could drastically improve it in several dimensions. Specifically, they could source a higher-res, higher frame rate, lighter, lower-latency OLED display, and connect it to rendering hardware with all the latency-reducing features John Carmack just described. These features aren't necessarily expensive from a hardware POV; it's just that nobody's seen the need to implement them yet.
  - neverm0re 14y ago
    > Specifically, they could source a higher-res, higher frame rate, lighter, lower-latency OLED display None of those are actually remotely as important as the fact you're still using your head as a dumb camera joystick, there's no ability to track eye movement and the mismatch between what you're seeing and your inner-ear pressure is going to make a number of people throw up.[1] These are fundamental problems that have been a part of HMDs for the last twenty years of me playing with them and the Rift addresses absolutely none of them. The Rift itself is not significantly different than its HMD contemporaries, it's actually aiming for the low-end gamer market and so far seems to be mostly pushing 'What's old is new again' without having made significant progress along the way. The most interesting part of the Rift kit is the tracker, not the actual HMD itself. [1]: In a study conducted by U.S. Army Research Institute for the Behavioral and Social Sciences in a report published May 1995 titled "Technical Report 1027 - Simulator Sickness in Virtual Environments", out of 742 pilot exposures from 11 military flight simulators, "approximately half of the pilots (334) reported post-effects of some kind: 250 (34%) reported that symptoms dissipated in less than 1 hour, 44 (6%) reported that symptoms lasted longer than 4 hours, and 28 (4%) reported that symptoms lasted longer than 6 hours. There were also 4 (1%) reported cases of spontaneously occurring flashbacks." Read up on 'Simulation Sickness' for more details.
    - modeless 14y ago
      Eye tracking would be awesome but it's not required for good VR. I fail to see how "using your head as a dumb camera joystick" is a problem; that's the whole point of VR. Motion sickness is a real problem for some people (but not all). With low latency and thoughtful game design I think it can be mitigated. The bigger problem is the social acceptability of blocking your entire field of vision for long periods of time. I won't pretend that VR doesn't have problems, but the payoff is large enough that the problems are worth tackling.
      - neverm0re 14y ago
        > I fail to see how "using your head as a dumb camera joystick" is a problem; that's the whole point of VR. Perhaps you should try playing some VR games for a while. I clocked quite a few hours playing the old Virtuality SU2000 games like Dactyl Nightmare and I've idly kept up with HMDs. The chief problem is that without eye tracking, it's incredibly UNFUN to use your head for movements that your eyes could otherwise have done for you. Mind you when the HMDs were much heavier back then it sucked a lot more, but it's still pretty shitty not being able to glance aside. Nope, gotta move your entire head for absolutely everything related to what you're currently seeing, if you move your head for any reason you can't maintain focus on objects naturally, etc. > Motion sickness is a real problem for some people (but not all). Probably not something you should underestimate. See Nintendo's 3DS launch and about-face on their stance on pushing 3D once they found a small but significant percentage of their users could not actually see the 3D effect. This lead to policy that the 3D effect could not be used for anything related to actual gameplay mechanics, reducing it entirely to an optional gimmick. Simulation sickness affects even more people than the 3D issue. It's a real problem if you want to go mainstream. Just to clarify my position, I'm not against the Rift nor do I have anything against HMDs. I simply see the Rift as a step along the way to whatever device truly popularizes the tech. I don't think we're there yet.
deleted 14y ago
[deleted]
untog 14y ago
I think VR is fundamentally broken, because you're going to try to react to things in a way that doesn't work. With a head-mounted display, the game can react as you move your head. Great. But then you'll instinctively try to walk towards something and it won't work (or you'll hit the wall). Same with picking things up, etc. It seems like it's deep in the uncanny valley of experience.
- WesleyJohnson 14y ago
  I think it's just a matter of time before the other pieces to the VR puzzle catch up. Omni-directional treadmills are already in existence and are a basic step towards allowing you to walk in a VR environment. Developments in prosthetics with 'feeling' could help pave way towards shoes, gloves or other garments that could provide you with the sensory experiences of walking on different surfaces or picking up in-game items.
  - untog 14y ago
    But at that point you've gone from a games console being something that takes up a small amount of space underneath your TV to something that takes up an entire room. From requiring you to pick up a controller to putting on an entire outfit. I'm just not sure the average consumer is in any way ready/willing for that.
    - purplelobster 14y ago
      I think the first successful VR you'll see will be closer to an arcade machine than a home console.
      - stcredzero 14y ago
        > I think the first successful VR you'll see will be closer to an arcade machine than a home console. I was just thinking about this. My development rig has 2 27" screens. If I added a 3rd one, this would cover most of my field of view horizontally. What if someone sold a console rig packaged with 3 screen and a mount? (Work with IKEA to produce a compatible cheap desk.) This kind of thing is already done and is awesome for vehicular games. (Racing, tank, aircraft...) A 3 screen rig also could still strike a kid as "awesome," so would draw money from the wallets of parents. Or, for arcade-type settings, how about a seat with a projected 360 dome, HOTAS controls, and motion simulation?
ChuckMcM 14y ago
I don't think they were a missed opportunity. In order for them to 'miss' they had to be 'possible' which they haven't been for a long time. Its only lately where various things have come together to get close. I have followed the 'head mounted display' technology closely since its early inception in the mid-90's as a means of providing military pilots with situational awareness. The first problem with head mounted displays was resolution. Both in color space and in pixel space. Early displays were monochrome (either green or red based on LED display drivers) and had a roughly 256 x 256 dot pitch. It was ok for targeting reticules and basic instrumentation display (like an artificial horizon). Early work also wanted transparency (look through) displays because there wasn't a way to display what you were looking at in enough fidelity to do both the visual field and the indicator field. Later a company called Colorado Micro Displays (CMD) came to market and started offering color, but 320 x 200 was the best resolution they could do. Higher resolution displays cost tens of thousands of dollars and were essentially hand crafted out of unobtainum. Direct retinal illumination displays attempted to get past strict display resolution issues (a friend of mine worked on one of the x-y positioning device for doing this, it had 500 angstrom repeatability! And was, as expected, insanely expensive) It really hasn't been until recently when DLP type systems at sufficient resolution to eliminate the x-y type beam director have come to market. Combined with better LED phosphors to bring the cost of an RGB display with better than VGA (640 x 480) resolution into something practical. The incessant push to higher resolution phone screens has created the capability to make opaque but high resolution screens which can enable something like the Oculus Rift type displays. That requirement stems from how far away from the eyes you have to put the screen and thus the torque moment that is applied to your head. There was a great system at NASA which used 10" displays but the 'head mounted' part was more like sticking your head into some weird steadycam kind of device. Finally there is the challenge of both high fidelity and high frequency head orientation technologies. Prior to about 2005 the best you could do was a laser gyro for motion and an accelerometer for inertial reference. Doing that at the necessary frequency (typically 1000 updates per second) didn't become cost effective until about 2009. Now however, many of the technologies have finally matured to the point where OR can be done in small quantities for perhaps $2K/unit. A 'game console' is struggles if it costs more then $300. So as a 'controller' option, and even as the 'whole console' option, these sorts of glasses are still about 3 - 5 years from hitting a price point that makes them the 'killer' peripheral. And of course because they aren't here yet, they can't really have 'missed' :-)
mtgx 14y ago
Well Oculus Rift should support everything from PC's to consoles. But I agree. Stuff like Oculus Rift is the true next-gen computing. The PS4 and next Xbox leave me pretty cold, especially when I already have a laptop that can already play all the latest games.
- nitrogen 14y ago
  I think the PS4's insane memory bandwidth (8GB of memory on a 170GB/sec GDDR5 bus) makes it a very interesting contender against PCs (dual-channel DDR3 maxes out at 34GB/s).
  - roc 14y ago
    Given the PS4 is using an off-the-shelf x86 processor, how long do you think it'll be until other OEMs are pitching laptops with these same CPUs that have GDDR5 controllers for main memory? How long until Intel matches? How long until basically every CPU supports it and every manufacturer has one or more such systems?
    - nitrogen 14y ago
      They may be using an off-the-shelf x86 core, but their actual CPU die may have all kinds of customizations on it (like the addition of a GDDR5 memory controller). AIUI GDDR5 memory is much more expensive than DDR3. So it may happen eventually, but I don't think it will be any time soon, and believe me, I really want PCs to be better than consoles in every way.
      - roc 14y ago
        It just seems to me that whatever they're doing with AMD, AMD is going to be able to farm out to other parties. Given that both MS and Sony are using AMD cores paired with GDDR5 memory controllers, it seems almost certain that there's nothing approaching an exclusivity contract to prevent AMD from now selling that bundle to any maker of gaming PCs or SteamBox builds or whatever else. And the current/historical cost differential between DDR and GDDR5 certainly seemed to be a considerable problem back when these specs were first rumored. But the following things suggest that's not likely to be an issue preventing it from hitting other gaming PCs: 1. Sony managed to spec 8GB of it into a box they simply can't afford to subsidize greatly at the outset. [1] 2. Sony and MS both need to know the cost of any expensive components will drop massively over the next couple years. 3. PC gamers readily and regularly accept higher prices for their machines than console makers could ever stomach. So I don't see it becoming common. Truly very few non-gamers would care about the performance delta. But it seems like a thing that will be on offer from multiple vendors, for feasible prices, before long. (Particularly given that AMD could really use any edge against Intel.) [1] People wrote off a lot of the rumors on that basis alone. If Sony is confident that it can not only afford GDDR5 for system memory, but that it can afford so much, there has to be a massive price cut incoming.

efdee 14y ago

I feel stupid.

MattDL 14y ago
Honestly I think the vast majority of people (even on HN) feel fairly stupid when you read something like that. It's an utter blessing that there are people such as Carmack in the industry.
- BystanderX 14y ago
  - who are willing/eager to share their research and thoughts with the general public. I imagine he had to push it a little to get Zenimax to allow it to be publicized, since he slipped that credit in there.
breckinloggins 14y ago
The view update part was the most confusing, but I tried to understand it this way: Imagine you had a magical game engine that rendered the entire world perfectly accurately for every point in space and direction a viewer could possibly be looking at. All you had to do was say: RenderGameFrameForEveryPossiblePoint(); ... // Who knows how much time viewerPosition = QuicklyGetViewerPosition(); TellTheGPUToShowWorldAccordingTo(viewerPosition); Well then, you could postpone calling those last two functions until the absolute last minute. This way you have very little or no movement of the viewer's head between when you read their position and when you show the view for that position. But naturally, RenderGameFrameForEveryPossiblePoint() is slightly out of bounds of current technology. A lot of what Carmack was discussing, as I understood it, was simulating this effect as closely as possible. The way to do that, it seems, is: StartRenderingGameFrameAccordingTo(lastViewerPosition); ... // Stuff viewerPosition = QuicklyGetViewerPosition(); viewMatrix = ComputeViewMatrixDelta(lastViewerPosition, viewerPosition); FinalizeGPUFrameRenderWithNewViewMatrix(viewMatrix); That final bit is just a perspective transformation of a bunch of rendering that was already computed and given to the GPU. But if the viewer moves too quickly, you can easily move somewhere in the world that wasn't actually rendered, or your perspective could shift such that and object that was once occluded is now visible, or vice versa. It seems a lot of the complexity is there. The last thing he talked about, time warping, seems to be a similar thing only it's scanline by scanline. So in effect you're saying "hey, video card and display, I know you're going to force me to draw a whole frame at once, so I'm going to give you a frame where each scanline gets rendered a little bit into the future according to where the player is moving." The effect on a monitor would probably look like a forward shear, but on an HMD (if done correctly), it would correct for the natural shear caused by having to "freeze frame" the viewer's perspective for one entire frame instead of just a scanline. Some of this may be woefully incorrect, but it was how I explained it to myself. Please correct anything that's wrong or overly simplified.

pdog 14y ago

With the proliferation of smartphones and projects like Google Glass and Oculus Rift taking off, I really think we'll start to see custom chipsets to improve the performance of augmented and virtual reality systems (battery life, lower latency, and more processing power).

networked 14y ago

A related article from Valve's Michael Abrash that may be a better introduction to the matter: http://blogs.valvesoftware.com/abrash/latency-the-sine-qua-non-of-ar-and-vr/ http://blogs.valvesoftware.com/abrash/latency-the-sine-qua-n... https://news.ycombinator.com/item?id=4985100 https://news.ycombinator.com/item?id=4985100

breckinloggins 14y ago

It sounds like we would benefit from GPU and display manufacturers providing a set of standard low-level control primitives so that people like Carmack who know what they are doing can really play around with the entire pipeline without having to worry about all the things the cards and the displays are doing behind their backs. For example, a GPU could have a set of standard settings with full buffering and all the other things that "help them win the framerate wars", but a developer should be able to turn all of that off when needed. It's the same with displays. LCD manufacturers could, I suppose, allow a modern day "CRT Mode 13h" where you just have scan lines mapped to memory buffers and whatever shows up in those buffers gets turned into a pixel as quickly as possible. Are there technical challenges preventing this from happening, or is it mainly inertia and lack of need from the current market?

niggler 14y ago
"a set of standard low-level control primitives ... Are there technical challenges preventing this from happening, or is it mainly inertia and lack of need from the current market?" It's easier to iterate faster without having to deal with a standardization committee. See OpenGL history for some context.
Scene_Cast2 14y ago
Regarding your last point: it's getting better. Modern TVs have "game mode", where latency is minimized. The latest LCD panels use embedded Display Port, eDP. The latest eDP standard introduces a frame buffer on the LCD itself, where you can just write deltas and tell it to "swap". I think this is quite similar to what you're proposing, in concept.
- breckinloggins 14y ago
  I found this [1], which seems to be a good introduction to the new features and also why eDP is poised to replace LVDS (which I, as a non hardware guy, had to deal with in a previous embedded device project and hope to never deal with again). [1] http://embedded-computing.com/articles/embedded-greater-display-efficiency/ http://embedded-computing.com/articles/embedded-greater-disp...

stcredzero 14y ago

> Updating the imagery in a head mounted display (HMD) based on a head tracking sensor is a subtly different challenge than most human / computer interactions. Doesn't the military already have this solved for head mounted displays for attack helicopters and 4++ generation fighter jets? Heck, they have augmented reality displays for that matter. EDIT: Many of these problems could be solved by putting an entire purpose-built gaming rig in the headset. There are nicely capable chips for mobile devices with lots of power in the GPU. Design such a system from the ground-up for low latency. Accelerometer and head-tracking inputs would be low-level interrupts, for example. I wonder if this is how the military contractors solved this?

knodi 14y ago
Yes, they have with very powerful hardware thats not very consumer wallet friendly.
- stcredzero 14y ago
  > very powerful hardware thats not very consumer wallet friendly. Yes, but the mobile landscape may well have changed this.
niggler 14y ago
"Doesn't the military already have this solved for head mounted displays for attack helicopters and 4++ generation fighter jets? " Far from it. There was a recall a few years ago over glitches in a fighter jet HMD. It's still an active area of research and no less than 5 different systems employed by the US military divisions

stesch 14y ago

Anyone any experience with a Wrap 1200VR from Vuzix?

Suan 14y ago

Misread as "Legacy Mitigation Strategies" - that would have been a good read as well.

dchichkov 14y ago

> Conventional computer interfaces are generally not as latency demanding as virtual reality, but sensitive users can tell the difference in mouse response down to the same 20 milliseconds or so, making it worthwhile to apply these techniques even in applications without a VR focus. I wish people would start treating text editors that they are developing this way. As hard real time systems. Don't care much about virtual reality, but I'm sick of text (and code) editors with unpredictable latency of basic operations [like typing a character, making find call, etc].

scott_s 14y ago
Minor quibble: you mean soft real time systems: http://en.wikipedia.org/wiki/Real-time_computing#Criteria_for_real-time_computing http://en.wikipedia.org/wiki/Real-time_computing#Criteria_fo...
monochromatic 14y ago
What editor are you using, and how old is your computer? I haven't had problems like this in years.
kragen 14y ago
Darius Bacon and I have spent some time trying to figure out a buffer data structure that can scale to large files while guaranteeing predictable latency for basic operations like typing a character. (What's "making find call"?) I think you mean "soft real time", though. A hard real-time system is one which is unusable if it fails to meet a deadline even once; a jet engine control system, say, where failing to meet a deadline could result in engine parts penetrating the fuselage. A "soft real time" system is one where failing to meet a deadline is a failure, but tolerable if infrequent; say, once every million keystrokes, or every 7 days of video play time.
stcredzero 14y ago
> I wish people would start treating text editors that they are developing this way. IDEs. Most of the things people do in modern IDEs were done in Smalltalk (albeit, with uglier interfaces and some manual steps) over a decade back. One of the differences, though, is that these things were mostly highly responsive, because they were bloat-free. (Can't speak for IBM VisualAge. Also, one could rightly complain that they were feature-poor in their stock configurations.) This is in stark contrast to other IDEs and app server frameworks, which would take forever to start up, or forever to restart, or forever to complete an operation. One example, Extract Method, was always instantaneous. It's still lugubrious even in modern IDEs like XCode4.5. The same rules that apply to Web Apps also apply to programmer's tools. Delays are yuck. Responsiveness is yum! I'm also a musician, and from what I've seen playing around in Audacity, 5 milliseconds is definitely inaudible, even to some of the most talented, golden-eared musicians I know. I suspect that 20 milliseconds is going to start to bug some people, though they won't be able to say why. Also, awareness of subtle modulations of very small variations in rhythm and how they affect the feel of music seems to be an indicator of intelligence and aesthetic awareness. So often, the difference between blah and great involves things that are subtle and hard to put your finger on.

ansible 14y ago

You could cut latency further if you were able to measure the muscle contractions in your neck area. Another option is smart eye tracking too. Generally if you are going to move your head in a particular direction, you'll likely start moving your eyes first.

sultezdukes 14y ago

is there anyboy smarter than carmacck..maybe sweeneye

cousin_it 14y ago

It's amazing how much time Carmack spends working around the "added cleverness" of systems created by people, rather than the inherent difficulties of the problem. These days it also seems true about programming in general.