8 ms·
Emulator Latency
- pjc50 10y agoI've just thought up a conceptually simple but tremendously difficult to implement and CPU-consuming way of working round this: avoid latency by seeing into the future. Modern GPUs give you a ton of parallel processing. Old gamepads are binary with a fairly limited number of buttons not all of which can be pressed at once by someone with two thumbs. The emulated RAM state is also fairly small - kilobytes. So, run lots of copies of the emulator. One for each possible change in button press status from the current state. At each frame (50/60Hz), look at the current actual inputs and pick (frame,20ms audio samples) from the available precomputed choices. Start calculating the next frames based on the winning version of the emulator state, and discard the rest. (This is effectively branch prediction at the macrostate level).
- larsiusprime 10y agoMentioned in the article: > There are magic tricks beyond this, such as emulating every possible input one frame into the future, to cut out a single frame of latency. But with only one controller, this would require higan to emulate up to 4096 simultaneous SNES systems and well ... higan just isn't that fast, sorry.
- pjc50 10y agoMissed that, it's near the bottom. :(
- larsiusprime 10y agoI do wonder how many states simpler systems like the GB or NES would have since they have fewer buttons, simpler audio, and less ram. Still probably a tough call -- for an accuracy-focused emulator that requires a lot of processing power this sort of solution might be out of reach for some years. You could probably do it on less accurate emulators, but then you're already relying on hacks & cheats so you might as well use other solutions for lowering input lag.
- byuu 10y agoI think it's just barely possible for the fastest NES and GB emulators to use this trick for one single frame of lag reduction. Here, you have eight inputs (assumes one player on the NES.) But there's a trick: up+down and left+right aren't physically possible due to a rocker in the D-pad. So as a whole, there are nine possible D-pad states instead of sixteen. With four buttons, that's sixteen possible states. So the total is 144 states. Now look at QuickNES, blargg's masterpiece NES emulator that runs fullspeed on a 66MHz PowerPC system. If you had the fastest, overclocked octacore CPU Intel makes, you may be able to pull off 144 instances of the emulation. Especially since you don't have to actually output the audio and video for all the missed predictions to the real hardware. Or perhaps more sanely ... most games don't have you using start+select while playing. I think people would be willing to accept a large jitter when pressing those two buttons. So now you're talking 36 instances, which starts to look very reasonable. At least, for emulators that are okay with sacrificing accuracy. The most popular NES emulators today require 800MHz (Nestopia), 1.6GHz (Nintendulator), and I'm not sure on puNES, but presumably just as high. 36 instances of those are unlikely. Still, this would be a fun experiment if anyone were willing! :D Of course, the end result is a 16ms lag reduction. And again, I need to stress ... I get that latency stacks, but ... you're really going to have a hard time even telling the difference between the two. Even if you're one of those people who can beat Dodonpachi or Ninja Gaiden or Battletoads on one life.
- mikeash 10y agoWhy does it need to emulate every possible input, rather than emulating a single path assuming that the controller state remains unchanged? You can only display one future frame to the user, after all, and that seems like the best one to display.
- pjc50 10y agoThe point would be to render the frame(s) before the input arrives, to reduce latency. You can only display one frame, but you don't know which frame until the user actually inputs something.
- mikeash 10y agoOooh, I see! I was thinking of running the simulation forward by N frames optimistically, and then fixing it up when something else comes in, but the idea here is just to precompute every possible result. Nifty!
- byuu 10y agoI realize a lot of the 4096 states are extremely unlikely (especially Up+Down or Left+Right ... not physically possible on an unmodified controller); but if the inputs you predict end up wrong, then you have no choice but to run the frame again normally. This is going to cause an extreme jittering where the input lag doubles for some frames. Imagine audio stuttering from a scratched CD, or a framerate that suddenly dips from 60fps to 30fps for a moment and then resumes. With input, the effect is going to be even more jarring. If you're going to do this, you absolutely can't have a miss. Ever.
- mikeash 10y agoYep, makes total sense to me now. I was thinking of optimistic prediction, not deterministic precomputation.
- deleted 10y ago[deleted]
- scottlamb 10y agoThe article talks about the OS audio buffering (for mixing streams from multiple applications) and application audio buffering (for mixing streams from within the application), each adding 10–40 ms of latency. Why is the application audio buffer necessary? Can't the application just send all its streams on to the OS separately to let the OS do all the mixing with a single-layer approach? Is there some unrealistically low bound on the number of possible streams or something that makes this impractical? Could that be fixed? This seems like a simple way to save 10–40ms, without giving up mixing audio across applications (as is necessary with the "WASAPI exclusive mode" the author described).
- haberman 10y agoWhat kind of "stream" did you have in mind? For the OS to do software mixing, it is basically going to run a "for" loop over the audio buffers and add them up to produce the audio buffer that goes to the hardware. When is it going to run that loop?
- nibnib 10y agoAn alternative is to do mixing on the sound card, but in that case you still need enough audio to fill up the hardware buffers.
- scottlamb 10y ago> What kind of "stream" did you have in mind? The same kind the author means. I didn't coin this term or introduce it to the discussion. > For the OS to do software mixing, it is basically going to run a "for" loop over the audio buffers and add them up to produce the audio buffer that goes to the hardware. When is it going to run that loop? Right, the author is saying that the OS is already doing this, and additionally the application does the same thing before sending a combined stream to the OS. So to answer your question of "when is [the OS] going to run that loop?" My answer is the same time it does now. I'm not proposing any changes to how the OS works. My question is: why does the application need to do that work? Why can't it send each stream to the OS to have them be combined in only one place?
- nibnib 10y agoGreat article. >This process can incur quite a bit of CPU time as well. Attempting to poll the keyboard state, mouse state, and all attached gamepads can easily eat several milliseconds per call. So it's just not possible to poll every millisecond. If input software layers mean it's not possible to poll every millisecond, why bother polling the hardware at 1kHz? Is it a just-in-case solution to increase hardware polling to maximum? I'm also curious if there are any harder numbers available, maybe by triggering a USB key input and measuring time for a test program to register the change. I know this sort of thing is done to compare CRTs and LCDs, I've never seen it done for a whole PC.
- byuu 10y agoThank you! I was really worried about the tone being too harsh or know-it-all. Wasn't really meant for audiences that weren't aware of the context. But as an emulator author, you get people presenting new zany latency reduction ideas that defy the laws of physics all the time, and they completely dismiss your own experience in the field, and it's like kids constantly saying, "are we there yet?" on a long car ride. Eventually you lose your cool, and then well ... you sound like me in that article >_> > why bother polling the hardware at 1kHz? Is it a just-in-case solution to increase hardware polling to maximum? Yes, pretty much. More of a because-we-can and to combat the cumulative effects of latency (death by a thousand papercuts) however possible. If you were to push it to 200Hz (5ms), then it becomes possible that your OS API returns states immediately after and it stacks with your emulator latency of 5ms to form a 10ms latency. Push it to 1000Hz and that drops to a 6ms maximum latency. It is indeed silly. No one is going to perceive a worst-case 4ms difference. (And I say worst-case because these sorts of misses tend to average between best-case and worst-case, so in practice it's probably half that bad.) We're trying to chase the emulation latency of a gamepad that you literally tell it exactly when to poll the inputs and within mere cycles on a 21MHz clock, start reading out the results from its shift-register. > I'm also curious if there are any harder numbers available That would be fun. I'll admit that many of the numbers are estimated. In the end, we can only observe the net total of all latency by pressing a button and seeing how quickly the sprites respond visually and aurally. But it's probably possible to isolate similar test cases for each source of latency. In a lot of cases (kernel audio mixing, keyboard responses), we're probably talking much smaller latencies than CRT vs LCD monitors, so you'd need a huge amount of precision.
- pjc50 10y agoThe only real way round this is not to do the emulation on a commodity multi-user system (with all its tremendous advantages!), but in some kind of hardware or bare metal software. Where you can access the original gamepads with the original sub-millisecond latency. Still can't quite work around the latency imposed by the display, unless you go back to CRT or build your own LVDS driver board. I had the opportunity to play a Tempest arcade machine recently, and the combination of low latency and the weighted spinning controller was really tangible.
- ilitirit 10y ago> Still can't quite work around the latency imposed by the display, unless you go back to CRT or build your own LVDS driver board. According to popular consensus answer was SED which eventually lost to LCD. https://en.wikipedia.org/wiki/Surface-conduction_electron-emitter_display https://en.wikipedia.org/wiki/Surface-conduction_electron-em...
- thescriptkiddie 10y agoOLED displays also have very low latency as long as there is no scaler in the way. The Oculus Rift CV1 uses them and has a motion-to-photons latency of 9 ms[0], which includes not just the display but the entire stack. https://www.reddit.com/r/oculus/comments/43p3a6/does_anybody_know_the_motiontophoton_latency_of/czjtj48 https://www.reddit.com/r/oculus/comments/43p3a6/does_anybody...
- byuu 10y agoYeah, scalers and OSDs are going to be the hardest things to give up. I will admit ... it's painful using my ZR30w with no scaler. I can't connect my PSP's component out to it, I can't hook up my Wii U, I can't connect my XRGB Mini, my Raspberry Pi, etc ... because none of them output at 2560x1600. Even worse, the DisplayPort connector stopped working on it (hooray, $1300 monitor quality!!), so all I have is one DVI port left, which can't even output at the 30-bit color depth which was a big part of why I wanted this monitor :(
- ilitirit 10y agoThere are well-known methods of significantly reducing latency (well, usually the methods are exclusive to certain emulators), but often the implication is that you're not sticking to faithful emulation of the hardware/software any more. FWIW: I think byuu overreacted to the last few questions regarding latency-related issues. http://board.byuu.org/phpbb3/viewtopic.php?f=8&t=1058&start=60 http://board.byuu.org/phpbb3/viewtopic.php?f=8&t=1058&start=... Apparently he won't even look at the test results... Besides all that, while I completely agree that the easiest way to reduce latency is by getting better hardware etc, in terms of competitive gaming 1f of lag is quite a big deal at high levels of play for some games. For example, games often utilize the concept of "just frame" inputs, meaning that inputs are required on very a specific frame (the definition has been slightly relaxed in recent years). Now there are two types of ways to do these: Muscle Memory (the easiest, you just repeat the move non stop in practice mode till it becomes second nature), and using an external cue (eg. audio/video). When it comes to muscle memory, input lag isn't that much of a deal breaker because once the first move is inputted, everything else will follow correctly, even if the entire sequence is 1f or 2f off. However, when it comes to using external cues, even 1f latency can cause the sequence to fail.
- stepvhen 10y agoI understand your argument, but I don't think anybody is playing SNES games competitively, on a non-negligable scale. Moreover if such a game exists, it would probably fall into the pathological cases he mentioned.
- ilitirit 10y agoThe thing of course is that the so-called minority are in fact the people who raise the issue of input lag in the first place. For example, the entire Speed Running community considers this an important topic. While they do play on "real" hardware, when runners practice for runs they generally load up a particular scenario in an emulator and repeat the practice a particular section that may require just frame inputs over and over until they are comfortable enough that it can be considered a valid strategy. So while it's true that the SR community only forms a minority of gamers, any sort of argument that relies on the experience of the majority effectively rules them out of a conversation that affects them the most. But besides that, it could be argued that in today's day and age, SNES Speed Runners probably do form a non-negligent subgroup, considering that the majority of gamers don't play games like SMW, Metroid or Megaman any more. For those who aren't aware, the Speed Running community has charity events that have in the past raised over $1m for cancer research etc. They may be "small", but they are far from insignificant in terms of their gaming presence. https://gamesdonequick.com/ https://gamesdonequick.com/
- gregpardo 10y agoBeen following byuu since my rom hacking days. I always stop by to read his articles and this is another good one.
- Zardoz84 10y agoWell, I never feel that ZSNES on DOS (on a 486) was unresponsive compared against my NES clone, when I was playing Super Mario 3 on NES and on ZSNES.
- deleted 10y ago[deleted]
- jdbernard 10y agoIn case you aren't trolling, the reason you are getting downvoted is probably because you are comparing apples to oranges. ZSNES is not an accurate emulator. byuu's goal is to emulate the systems perfectly. Here is an article he wrote about it: http://arstechnica.com/gaming/2011/08/accuracy-takes-power-one-mans-3ghz-quest-to-build-a-perfect-snes-emulator/ http://arstechnica.com/gaming/2011/08/accuracy-takes-power-o... So yes, ZSNES feels a lot more responsive than higan. But that's because it cuts a lot of corners with regards to accuracy of emulation. It is probably the least accurate emulator for this reason. It prioritizes speed. In light of that, your comment doesn't really add anything to the discussion.
- makomk 10y agoPerhaps the delay from user input to actual action on the screen should be considered when we talk about how accurate emulators are. After all, it's something that can noticeably affect how games actually play.
- byuu 10y agoIf you can run higan at 100% speed, then ZSNES has more lag than higan does (due to double buffering, polling only once per frame, etc.) For me, the issue with the OP is a) comparing to clone NES hardware, and b) it's subjective. Different people react differently to latency. The numbers I talk about may be largely estimated, but they're objectively real latencies that really do exist, even if you can't observe them personally. It's a very good thing if you can't. Makes gaming a lot more fun. Testing myself, I don't really detect a change in latency under emulation alone until I simulate adding about 75ms more than is already there. However, I did notice a problem when I moved from playing Ninja Gaiden Trilogy (I know, bad port) on my SNES to higan: all of my timed moves were failing (it's a game that requires pixel-perfect movements); but I adapted pretty quickly and was able to beat the game. But again, this is all subjective stuff, so it's not really adding to the technical discussion any.
- nwmcsween 10y agoIf the input is steady could you not do some sort of predictive rendering where a frame or more is prerendered?