4 ms·

The article talks about the OS audio buffering (for mixing streams from multiple applications) and application audio buffering (for mixing streams from within t

The article talks about the OS audio buffering (for mixing streams from multiple applications) and application audio buffering (for mixing streams from within the application), each adding 10–40 ms of latency. Why is the application audio buffer necessary? Can't the application just send all its streams on to the OS separately to let the OS do all the mixing with a single-layer approach? Is there some unrealistically low bound on the number of possible streams or something that makes this impractical? Could that be fixed? This seems like a simple way to save 10–40ms, without giving up mixing audio across applications (as is necessary with the "WASAPI exclusive mode" the author described).

haberman 10y ago
What kind of "stream" did you have in mind? For the OS to do software mixing, it is basically going to run a "for" loop over the audio buffers and add them up to produce the audio buffer that goes to the hardware. When is it going to run that loop?
- nibnib 10y ago
  An alternative is to do mixing on the sound card, but in that case you still need enough audio to fill up the hardware buffers.
- scottlamb 10y ago
  > What kind of "stream" did you have in mind? The same kind the author means. I didn't coin this term or introduce it to the discussion. > For the OS to do software mixing, it is basically going to run a "for" loop over the audio buffers and add them up to produce the audio buffer that goes to the hardware. When is it going to run that loop? Right, the author is saying that the OS is already doing this, and additionally the application does the same thing before sending a combined stream to the OS. So to answer your question of "when is [the OS] going to run that loop?" My answer is the same time it does now. I'm not proposing any changes to how the OS works. My question is: why does the application need to do that work? Why can't it send each stream to the OS to have them be combined in only one place?
  - haberman 10y ago
    Ah, I see what you are asking. Sorry I misunderstood. I doubt that the application mixing is actually adding any latency compared to if the application only had a single logical stream internally. Regardless of the application's internal structure, you will at every moment have a set of audio buffers: 1. the one the sound card is currently playing 2. the one the OS is preparing for the sound card (it gets one buffer's worth of time to synthesize this). 3. the one the application is preparing for the OS (it gets one buffer's worth of time to synthesize this). If the OS isn't doing any mixing, then you could make buffer 2 and 3 the same and save a buffer's worth of latency. But if the OS is doing mixing, then it needs a chance to add up all the application buffers before they go to the sound card (that is its "synthesize" step). So the application can't be writing directly into the OS's buffer. You might ask "why can't the application and OS both do their work within a single buffer's worth of time? Hmm, I guess it is an interesting question whether the OS could, inside the write() call, do the mixing immediately. I'm reaching the point where I'd have to speculate: I'm not exactly sure how existing OSs design their mixing and whether this would be feasible or not.
  - byuu 10y ago
    > My question is: why does the application need to do that work? Why can't it send each stream to the OS to have them be combined in only one place? Two reasons. One, the audio API used is a simple push buffer design (eg OSS) and so you have to do your own mixing first. XAudio2 and friends will spare you this detail. Two, you're emulating a piece of gaming hardware that has its own internal mixer. And when that internal mixer doesn't have enough sound channels, the video games running on said emulated system can include their own software mixers to make it even more fun.
- byuu 10y ago
  Good catch ... I didn't account for that in my article. In emulation, an audio stream is an infinitely long queue of audio samples, for which you can only have a certain amount of them known ahead of time (and that amount is itself a source of audio latency.) You usually have one, but can sometimes have more. And they can vary in their frequencies, including being pathological (the Game Boy outputs sound at 2MHz.) In PC gaming, you probably have some kind of looping for background music; but sound effects are definitely things you can push in the smarter audio APIs (like XAudio2) all at once and have all the data available to consume whenever the audio API->OS mixer->sound card needs the data.
aji 10y ago
this is a really good question, and the more I think about it, the less I'm able to come up with an answer assuming each in-application sound generator is pull-based (like the OS sound APIs typically are) surely each one could write to a single accumulator buffer which is then sent to the OS as-is, eliminating the need for any special mixing I'd like to know more about the requirements that led to this extra in-application mixing latency edit: obviously the presence of push-based sound generation in the application would add another layer of mixing and latency, requiring previously prepared buffers to be mixed into a signal for the OS milliseconds after the source buffers were initially filled. however, this would be a self-imposed limitation that I still don't see a reason for
byuu 10y ago
> Why is the application audio buffer necessary? It's more about the APIs used. With something like OSS, you just write samples to a (virtualized to many devices) /dev/dsp handle. So you have to take your background music, and each sound effect channel, and mix the values together. But if you're writing a PC game on Windows, you'd open separate IXAudio2SourceVoice handles for each, and I'm presuming that XAudio2 will handle all the mixing and when to optimally push samples to the hardware for you. You're working at a higher level here when you can provide the hardware/OS audio API with more information. But an emulator just gets a solid stream of pre-mixed samples. Sometimes the game software have their own mixers (like Tales of Phantasia on the SNES), and these tend to fall back on game console hardware mixers, that then pass a rendered stream of audio out to the speakers (hence, indirection.) With audio, how you handle it depends on the backend API. With DirectSound and WASAPI exclusive, it acts like a ring buffer in RAM that loops around and you poll the cursor (sample) position to know when and where to write. With XAudio2 and WASAPI shared, you build up tiny blocks of samples and throw them at the sound API. In both cases, even at 44KHz, it's not a smart idea to do this for every single sample output. So you're going to have a software queue that accumulates these samples. And the size of these queues aren't just about minimizing calls to the sound API, but it's also about the API's own precision and at what resolution it can report when a pushed buffer is consumed or where the cursor is at. This varies wildly between PCs. So the best you can really do is offer the user a latency selection and let them see how low they can set the value without audio distortion from misses. A final complexity with emulation is that we can still have multiple audio streams. When you plug a Super Game Boy cart into an SNES, or a CD attachment onto a Genesis; the emulation of these usually results in two streams. As mentioned in the article, generally you can merge two streams in software in near-realtime. Presuming both sound sources run at the same frequency, you only need one sample queued from both sources before you can mix them and send them to your sound API output buffer. Though keep in mind we are making a parallel process serial, and the context switches between components is the biggest source of slowdown in all of emulation, so sometimes you'll have quite a few samples from one component queuing up before we get any from another. It gets a bit fancier when they vary frequencies (and they do in emulation) ... and then you get to pull in anti-aliasing, resampling audio filters (I use butterworth biquad type-2 IIR with cubic resampling in higan), so the queues have to get a little tiny bit larger. On anti-aliasing, that's yet more fun ... a lot of these old game consoles produced sound in the megahertz range, so you have to strip out the frequencies above the Nyquist limit to prevent aliasing (annoying buzzing) when you resample it to rates PCs can handle (in the kilohertz range.)
squeaky-clean 10y ago
It's much more efficient to work on an audio buffer than to do it one sample at a time, and the larger the buffer the more efficient. An analogy would be sending 500 POST requests with a single JSON object as the body, or a single POST request containing an JSON array of 500 objects. Higan does allow you to reduce the application latency. But they make the point many times in the webpage that doing so can cause audio glitches, pops and distortion due to buffer underruns. My guess is they want to settle on the best default settings. Something that won't have a noticeable latency to most players, but also won't provide audio glitching when played on, say, an older laptop.

haberman 10y ago

What kind of "stream" did you have in mind? For the OS to do software mixing, it is basically going to run a "for" loop over the audio buffers and add them up to produce the audio buffer that goes to the hardware. When is it going to run that loop?

nibnib 10y ago
An alternative is to do mixing on the sound card, but in that case you still need enough audio to fill up the hardware buffers.
scottlamb 10y ago
> What kind of "stream" did you have in mind? The same kind the author means. I didn't coin this term or introduce it to the discussion. > For the OS to do software mixing, it is basically going to run a "for" loop over the audio buffers and add them up to produce the audio buffer that goes to the hardware. When is it going to run that loop? Right, the author is saying that the OS is already doing this, and additionally the application does the same thing before sending a combined stream to the OS. So to answer your question of "when is [the OS] going to run that loop?" My answer is the same time it does now. I'm not proposing any changes to how the OS works. My question is: why does the application need to do that work? Why can't it send each stream to the OS to have them be combined in only one place?
- haberman 10y ago
  Ah, I see what you are asking. Sorry I misunderstood. I doubt that the application mixing is actually adding any latency compared to if the application only had a single logical stream internally. Regardless of the application's internal structure, you will at every moment have a set of audio buffers: 1. the one the sound card is currently playing 2. the one the OS is preparing for the sound card (it gets one buffer's worth of time to synthesize this). 3. the one the application is preparing for the OS (it gets one buffer's worth of time to synthesize this). If the OS isn't doing any mixing, then you could make buffer 2 and 3 the same and save a buffer's worth of latency. But if the OS is doing mixing, then it needs a chance to add up all the application buffers before they go to the sound card (that is its "synthesize" step). So the application can't be writing directly into the OS's buffer. You might ask "why can't the application and OS both do their work within a single buffer's worth of time? Hmm, I guess it is an interesting question whether the OS could, inside the write() call, do the mixing immediately. I'm reaching the point where I'd have to speculate: I'm not exactly sure how existing OSs design their mixing and whether this would be feasible or not.
- byuu 10y ago
  > My question is: why does the application need to do that work? Why can't it send each stream to the OS to have them be combined in only one place? Two reasons. One, the audio API used is a simple push buffer design (eg OSS) and so you have to do your own mixing first. XAudio2 and friends will spare you this detail. Two, you're emulating a piece of gaming hardware that has its own internal mixer. And when that internal mixer doesn't have enough sound channels, the video games running on said emulated system can include their own software mixers to make it even more fun.
byuu 10y ago
Good catch ... I didn't account for that in my article. In emulation, an audio stream is an infinitely long queue of audio samples, for which you can only have a certain amount of them known ahead of time (and that amount is itself a source of audio latency.) You usually have one, but can sometimes have more. And they can vary in their frequencies, including being pathological (the Game Boy outputs sound at 2MHz.) In PC gaming, you probably have some kind of looping for background music; but sound effects are definitely things you can push in the smarter audio APIs (like XAudio2) all at once and have all the data available to consume whenever the audio API->OS mixer->sound card needs the data.

aji 10y ago

this is a really good question, and the more I think about it, the less I'm able to come up with an answer assuming each in-application sound generator is pull-based (like the OS sound APIs typically are) surely each one could write to a single accumulator buffer which is then sent to the OS as-is, eliminating the need for any special mixing I'd like to know more about the requirements that led to this extra in-application mixing latency edit: obviously the presence of push-based sound generation in the application would add another layer of mixing and latency, requiring previously prepared buffers to be mixed into a signal for the OS milliseconds after the source buffers were initially filled. however, this would be a self-imposed limitation that I still don't see a reason for

byuu 10y ago

> Why is the application audio buffer necessary? It's more about the APIs used. With something like OSS, you just write samples to a (virtualized to many devices) /dev/dsp handle. So you have to take your background music, and each sound effect channel, and mix the values together. But if you're writing a PC game on Windows, you'd open separate IXAudio2SourceVoice handles for each, and I'm presuming that XAudio2 will handle all the mixing and when to optimally push samples to the hardware for you. You're working at a higher level here when you can provide the hardware/OS audio API with more information. But an emulator just gets a solid stream of pre-mixed samples. Sometimes the game software have their own mixers (like Tales of Phantasia on the SNES), and these tend to fall back on game console hardware mixers, that then pass a rendered stream of audio out to the speakers (hence, indirection.) With audio, how you handle it depends on the backend API. With DirectSound and WASAPI exclusive, it acts like a ring buffer in RAM that loops around and you poll the cursor (sample) position to know when and where to write. With XAudio2 and WASAPI shared, you build up tiny blocks of samples and throw them at the sound API. In both cases, even at 44KHz, it's not a smart idea to do this for every single sample output. So you're going to have a software queue that accumulates these samples. And the size of these queues aren't just about minimizing calls to the sound API, but it's also about the API's own precision and at what resolution it can report when a pushed buffer is consumed or where the cursor is at. This varies wildly between PCs. So the best you can really do is offer the user a latency selection and let them see how low they can set the value without audio distortion from misses. A final complexity with emulation is that we can still have multiple audio streams. When you plug a Super Game Boy cart into an SNES, or a CD attachment onto a Genesis; the emulation of these usually results in two streams. As mentioned in the article, generally you can merge two streams in software in near-realtime. Presuming both sound sources run at the same frequency, you only need one sample queued from both sources before you can mix them and send them to your sound API output buffer. Though keep in mind we are making a parallel process serial, and the context switches between components is the biggest source of slowdown in all of emulation, so sometimes you'll have quite a few samples from one component queuing up before we get any from another. It gets a bit fancier when they vary frequencies (and they do in emulation) ... and then you get to pull in anti-aliasing, resampling audio filters (I use butterworth biquad type-2 IIR with cubic resampling in higan), so the queues have to get a little tiny bit larger. On anti-aliasing, that's yet more fun ... a lot of these old game consoles produced sound in the megahertz range, so you have to strip out the frequencies above the Nyquist limit to prevent aliasing (annoying buzzing) when you resample it to rates PCs can handle (in the kilohertz range.)

squeaky-clean 10y ago

It's much more efficient to work on an audio buffer than to do it one sample at a time, and the larger the buffer the more efficient. An analogy would be sending 500 POST requests with a single JSON object as the body, or a single POST request containing an JSON array of 500 objects. Higan does allow you to reduce the application latency. But they make the point many times in the webpage that doing so can cause audio glitches, pops and distortion due to buffer underruns. My guess is they want to settle on the best default settings. Something that won't have a noticeable latency to most players, but also won't provide audio glitching when played on, say, an older laptop.