3 ms·

ALSA is not the problem, it works very fine for the low-latency case, I can reliably run soundcards with light processing load at 96 kHz @ 64 frames/period (20.

by cnvogel 12y ago

ALSA is not the problem, it works very fine for the low-latency case, I can reliably run soundcards with light processing load at 96 kHz @ 64 frames/period (20.7ms --> 1.4 ms latency) on a quad core i5, e.g. for running a reverb effect, but most of the time I only record and will settle for 1024 frames/period or so. (210ms --> 20ms). The period size, just for completeness, is the number of samples recorded on each block that is forwarded to the audio processing application. If whatever audio framework you use doesn't allow to run processing with a input-to-output delay (latency) of two times the period size, it's broken (probably the case for Audio Flinger at Android, don't know much about it). ➜ ~ jackd -d alsa -p 64 -r 96000 jackdmp 1.9.10 Copyright 2001-2005 Paul Davis and others. Copyright 2004-2014 Grame. (...) creating alsa driver hw:0|hw:0|64|2|96000|0|0|nomon|swmeter|-|32bit configuring for 96000Hz, period = 64 frames (0.7 ms), buffer = 2 periods ALSA: final selected sample format for capture: 32bit integer little-endian ALSA: use 2 periods for capture ALSA: final selected sample format for playback: 32bit integer little-endian ALSA: use 2 periods for playback (this is on my laptop, just for illustration purposes)

jblow 12y ago
But ... why is there a period size? Isn't that a broken design that can only introduce latency? What is wrong with "however much audio data is ready when the application asks, send it"?
- cnvogel 12y ago
  Well... it's like all audio chipsets work nowadays. Only some DSPs will be able to efficiently handle single-frame data processing, but they have the help of dedicated address generators and lightweight interrupts synchronized to the digital interface. If you write "process however much audio data is ready", then you already imply that your CPU will not be up to speed to process 48000 interrupts/second reliably and you need some buffering. And if you have to assume that sometimes you'll miss 100 samples (which, then, you'll process en-block), this means that to work reliably, you'll have to start at least 100 samples early so that you don't miss the deadline of the DAC, because the DAC will, with intractably output one sample every 48000th of a second. This already implies some kind of periodic processing of blocks, doesn't it? (and yes, such a scheme will theoretically allow you to half the latency from something to 2period-size to 1period-size + the time for processing) Third, a lot of the algorithms for processing audio can be implemented much more efficiently if you have a known block size and don't have to calculate your filters or convolutions with constantly changing number of samples for every step. Also efficiency of processing will decrease (reloading the cache after each interrupt when switching from processing plugin to processing plugin), so the time spent on calculating per frame will go up if your period size gets smaller. At one point you'll need exactly one "period size" to calculate one period size worth of samples: That's the maximum your machine can handle, and at that point you'll have a latency of your "period size"*2, which is exactly the same as running with a fixed period-size ;-). And as you can choose the period-size rather freely (maybe completely arbitrary, maybe 2^n, depends on the chipset/hardware) there's no disadvantage left.
  - jblow 12y ago
    As someone who has actually done a good amount of soft-real-time audio programming, I can tell that you probably haven't. Everything you are saying about CPU speeds is made-up nonsense. Look into how these things are done on systems where folks actually care about latency (for example, commercial audio hardware, game consoles, etc).
    - cnvogel 12y ago
      Please tell me what claim I made about CPU speeds is unsubstantiated nonsense?
      - jblow 12y ago
        I could type up a thorough explanation, but it would take about an hour, and I have a lot to do. It is actually not a bad idea to do such a write-up, but I don't think the appropriate venue for it is an ephemeral post on Hacker News ... I'd rather blog it somewhere that's more suitable for long-term reference. But I'll drop a few hints. First of all, nobody is talking about running interrupts at 48kHz. That is complete nonsense. The central problem to solve is that you have two loops running and they need to be coordinated: the hardware is running in a loop generating samples, and the software is running in a (much more complicated) loop consuming samples. The question is how to coordinate the passing of data between these with minimal latency and maximum flexibility. If you force things to fill fixed-size buffers before letting the software see them (say, 480 samples or whatever), then it is easy to see problems with latency and variance: simply look at a software loop with some ideal fixed frame time T and look at what happens when T is not 100Hz. (Let's say it is a hard 60Hz, such as on a current game console). See what happens in terms of latency and variance when the hardware is passing you packets every 10ms and you are asking for them every 16.7ms. The key is to remove one of these fixed frequencies so that you don't have this problem. Since the one coming from the hardware is completely fictitious, that is the one to remove. Instead of pushing data to the software every 10ms, you let the software pull data at whatever rate it is ready to handle that data, thus giving you a system with only one coarse-grained component, which minimizes latency. You are not running interrupts at 48kHz or ten billion terahertz, you are running them exactly when the application needs them, which in this case is 16.7ms (but might be 8.3ms or 10ms or a variable frame rate). You don't have to recompute any of the filters in your front-end software based on changing amounts of data coming in from the driver. The very suggestion is nonsense; if you are doing that, it is a clear sign that your audio processing is terrible because there is a dependency between chunk size and output data. It should be obvious that your output should be a function of the input waveform only. To achieve this, you just save up old samples after you have played them, and run your filter over those plus the new samples. None of this has anything to do with what comes in from the driver when and how big. Edit: I should point out, by the way, that this extends to purely software-interface issues. Any audio issue where the paradigm is "give the API a callback and it will get called once in a while with samples" is terrible for multiple reasons, at least one of which is explained above. I talked to the SDL guys about this and to their credit they saw the problem immediately and SDL2 now has an application-pull way to get samples (I don't know how well it is supported on various platforms, or whether it is just a wrapper over the thread thing though, which would be Not Very Good.)
- fl0wenol 12y ago
  1) Userspace sleeps until audio data is available because there's an eternity of clock cycles between each sample; do you want to be woken up after every sample if you're done doing work or a handful? You could also busy wait but that kills battery life. 2) In order to hand you off some samples you have to at least make one copy. It's convenient to be able to copy an entire _something_ without worrying about the sound card trying to DMA into it (and any hardware-specific details to make that possible). The difference here is that with something like jackd (similar conceptually to CoreAudio or ASIO) there is just the hardware buffer in the kernel and the user buffer in jack which can be basically "shared" by all jack-enabled apps without additional copying on the user-space side. On the other hand you can't do sample-rate conversion and per-app volume control with something like that. But if you're doing audio software you're not worrying if flash is too loud and Skype is too soft. It's a whole-system thing enabled by the software and all the buffers and latency are being managed there.
  - jblow 12y ago
    Why is (2) dependent on anything regarding the number of samples you get at once? Sure, suppose there is a maximum block size; why does anything regarding copying "an entire something" require you to have filled that entire block size with live data? Why can't you just copy however much is available in the buffer? I don't understand why copies are even relevant: you can make several extra copies and nobody will ever notice. Audio data is trivial in modern systems. Let's say there are two channels coming in; 48000 * 2 * 2 bytes per second is an absolutely trivial amount of data to copy and has been for many years. Building some convoluted (and unreliable) system just to prevent one copy per application, when each application is going to be doing a lot of nontrivial processing on that data, strikes me as foolish. But don't listen to me, look at the fact that Linux audio is still famously unreliable. If the way it's done were a good idea, it would actually work and everyone would be happy with it.
    - FigBug 12y ago
      Typically because the hardware delivers audio in blocks. I know mostly about how USB works, but I imagine it applies to other hardware types as well. USB has several transfer types: interrupt, bunk and isochronous. Interrupt is initiated by the device so it wouldn't help you read variable amounts. Bulk is good for mass data transfers, but has guaranteed access to the bus, so you'd possible get audio dropouts when accessing another USB device. Isochronous can reserve bus bandwidth and have latency guarantees, but must occur on a fixed schedule. Since they are on a fixed schedule, they always have the same amount of data, hence a fixed block size. Since data is arriving at the OS in fixed blocks, the lowest latency way to handle the data is deal with the blocks when the arrive. If you wanted to read variable amounts of data, you'd need to add a buffer on top that could hold a variable amount of data which would add latency. Copying variable amounts of data isn't slow, but dealing with dynamic memory allocation is. If you needed to allocate a different sized block every few milliseconds you'd be spending the majority of your time allocating memory rather than processing audio. This system can work very well: OS X, iOS, Windows and Linux can all get very low latencies. The issues with Android has nothing to do with block sizes, but something else in it's architecture.
- anon4 12y ago
  This is actually how PulseAudio attempts to work. It doesn't always work out (skype's startup sound being infamously jittery), but the thought is there.

by cnvogel 12y ago

jblow 12y ago

But ... why is there a period size? Isn't that a broken design that can only introduce latency? What is wrong with "however much audio data is ready when the application asks, send it"?

cnvogel 12y ago
Well... it's like all audio chipsets work nowadays. Only some DSPs will be able to efficiently handle single-frame data processing, but they have the help of dedicated address generators and lightweight interrupts synchronized to the digital interface. If you write "process however much audio data is ready", then you already imply that your CPU will not be up to speed to process 48000 interrupts/second reliably and you need some buffering. And if you have to assume that sometimes you'll miss 100 samples (which, then, you'll process en-block), this means that to work reliably, you'll have to start at least 100 samples early so that you don't miss the deadline of the DAC, because the DAC will, with intractably output one sample every 48000th of a second. This already implies some kind of periodic processing of blocks, doesn't it? (and yes, such a scheme will theoretically allow you to half the latency from something to 2period-size to 1period-size + the time for processing) Third, a lot of the algorithms for processing audio can be implemented much more efficiently if you have a known block size and don't have to calculate your filters or convolutions with constantly changing number of samples for every step. Also efficiency of processing will decrease (reloading the cache after each interrupt when switching from processing plugin to processing plugin), so the time spent on calculating per frame will go up if your period size gets smaller. At one point you'll need exactly one "period size" to calculate one period size worth of samples: That's the maximum your machine can handle, and at that point you'll have a latency of your "period size"*2, which is exactly the same as running with a fixed period-size ;-). And as you can choose the period-size rather freely (maybe completely arbitrary, maybe 2^n, depends on the chipset/hardware) there's no disadvantage left.
- jblow 12y ago
  As someone who has actually done a good amount of soft-real-time audio programming, I can tell that you probably haven't. Everything you are saying about CPU speeds is made-up nonsense. Look into how these things are done on systems where folks actually care about latency (for example, commercial audio hardware, game consoles, etc).
  - cnvogel 12y ago
    Please tell me what claim I made about CPU speeds is unsubstantiated nonsense?
    - jblow 12y ago
      I could type up a thorough explanation, but it would take about an hour, and I have a lot to do. It is actually not a bad idea to do such a write-up, but I don't think the appropriate venue for it is an ephemeral post on Hacker News ... I'd rather blog it somewhere that's more suitable for long-term reference. But I'll drop a few hints. First of all, nobody is talking about running interrupts at 48kHz. That is complete nonsense. The central problem to solve is that you have two loops running and they need to be coordinated: the hardware is running in a loop generating samples, and the software is running in a (much more complicated) loop consuming samples. The question is how to coordinate the passing of data between these with minimal latency and maximum flexibility. If you force things to fill fixed-size buffers before letting the software see them (say, 480 samples or whatever), then it is easy to see problems with latency and variance: simply look at a software loop with some ideal fixed frame time T and look at what happens when T is not 100Hz. (Let's say it is a hard 60Hz, such as on a current game console). See what happens in terms of latency and variance when the hardware is passing you packets every 10ms and you are asking for them every 16.7ms. The key is to remove one of these fixed frequencies so that you don't have this problem. Since the one coming from the hardware is completely fictitious, that is the one to remove. Instead of pushing data to the software every 10ms, you let the software pull data at whatever rate it is ready to handle that data, thus giving you a system with only one coarse-grained component, which minimizes latency. You are not running interrupts at 48kHz or ten billion terahertz, you are running them exactly when the application needs them, which in this case is 16.7ms (but might be 8.3ms or 10ms or a variable frame rate). You don't have to recompute any of the filters in your front-end software based on changing amounts of data coming in from the driver. The very suggestion is nonsense; if you are doing that, it is a clear sign that your audio processing is terrible because there is a dependency between chunk size and output data. It should be obvious that your output should be a function of the input waveform only. To achieve this, you just save up old samples after you have played them, and run your filter over those plus the new samples. None of this has anything to do with what comes in from the driver when and how big. Edit: I should point out, by the way, that this extends to purely software-interface issues. Any audio issue where the paradigm is "give the API a callback and it will get called once in a while with samples" is terrible for multiple reasons, at least one of which is explained above. I talked to the SDL guys about this and to their credit they saw the problem immediately and SDL2 now has an application-pull way to get samples (I don't know how well it is supported on various platforms, or whether it is just a wrapper over the thread thing though, which would be Not Very Good.)
fl0wenol 12y ago
1) Userspace sleeps until audio data is available because there's an eternity of clock cycles between each sample; do you want to be woken up after every sample if you're done doing work or a handful? You could also busy wait but that kills battery life. 2) In order to hand you off some samples you have to at least make one copy. It's convenient to be able to copy an entire _something_ without worrying about the sound card trying to DMA into it (and any hardware-specific details to make that possible). The difference here is that with something like jackd (similar conceptually to CoreAudio or ASIO) there is just the hardware buffer in the kernel and the user buffer in jack which can be basically "shared" by all jack-enabled apps without additional copying on the user-space side. On the other hand you can't do sample-rate conversion and per-app volume control with something like that. But if you're doing audio software you're not worrying if flash is too loud and Skype is too soft. It's a whole-system thing enabled by the software and all the buffers and latency are being managed there.
- jblow 12y ago
  Why is (2) dependent on anything regarding the number of samples you get at once? Sure, suppose there is a maximum block size; why does anything regarding copying "an entire something" require you to have filled that entire block size with live data? Why can't you just copy however much is available in the buffer? I don't understand why copies are even relevant: you can make several extra copies and nobody will ever notice. Audio data is trivial in modern systems. Let's say there are two channels coming in; 48000 * 2 * 2 bytes per second is an absolutely trivial amount of data to copy and has been for many years. Building some convoluted (and unreliable) system just to prevent one copy per application, when each application is going to be doing a lot of nontrivial processing on that data, strikes me as foolish. But don't listen to me, look at the fact that Linux audio is still famously unreliable. If the way it's done were a good idea, it would actually work and everyone would be happy with it.
  - FigBug 12y ago
    Typically because the hardware delivers audio in blocks. I know mostly about how USB works, but I imagine it applies to other hardware types as well. USB has several transfer types: interrupt, bunk and isochronous. Interrupt is initiated by the device so it wouldn't help you read variable amounts. Bulk is good for mass data transfers, but has guaranteed access to the bus, so you'd possible get audio dropouts when accessing another USB device. Isochronous can reserve bus bandwidth and have latency guarantees, but must occur on a fixed schedule. Since they are on a fixed schedule, they always have the same amount of data, hence a fixed block size. Since data is arriving at the OS in fixed blocks, the lowest latency way to handle the data is deal with the blocks when the arrive. If you wanted to read variable amounts of data, you'd need to add a buffer on top that could hold a variable amount of data which would add latency. Copying variable amounts of data isn't slow, but dealing with dynamic memory allocation is. If you needed to allocate a different sized block every few milliseconds you'd be spending the majority of your time allocating memory rather than processing audio. This system can work very well: OS X, iOS, Windows and Linux can all get very low latencies. The issues with Android has nothing to do with block sizes, but something else in it's architecture.
anon4 12y ago
This is actually how PulseAudio attempts to work. It doesn't always work out (skype's startup sound being infamously jittery), but the thought is there.