7 ms·

(2011) But a great summary and mostly evergreen One practical reality it doesn't share is that your audio processing (or generation) code is often going to be

(2011) But a great summary and mostly evergreen One practical reality it doesn't share is that your audio processing (or generation) code is often going to be running in a bus shared by a ton of other modules and so you don't have the luxury of using "5.6ms" as your deadline for a 5.6ms buffer. Your responsibility, often, is to just get as performant as reasonably possible so that everything on the bus can be processed in those 5.6ms. The pressure is usually much higher than the buffer length suggests.

spacechild1 2y ago
What do you mean by "bus" and "module" in this context?
- GrantMoyer 2y ago
  A module is a piece of software or hardware which is independent in some way. A bus is a shared medium of communication[1]. Often, busses are time-division multiplexed[2], so if you want to use the bus, but another module is already using it, you need to wait. For example, if your audio buffers are ultimately submitted to a sound card over a PCI bus, the submission may need to wait for any ongoing transactions on the PCI bus, such as messages to a graphics card. [1]: https://en.wikipedia.org/wiki/Bus_(computing) https://en.wikipedia.org/wiki/Bus_(computing) [2]: https://en.wikipedia.org/wiki/Time-division_multiplexing https://en.wikipedia.org/wiki/Time-division_multiplexing
  - spacechild1 2y ago
    That is one possible interpretation, but not what they meant. That's why I asked because I wasn't sure :)
- swatcoder 2y ago
  "Bus" (as I was using it) is the path from some audio source to some audio destination and a "module" (as used) would be something that takes a buffer of samples on that bus and does something with it. You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In this case, you'd more or less control the whole bus and all the code processing samples on it and have a fairly true sense of your deadline. (The OS and drivers would still be introducing some overhead for mixing or resampling, etc, but that's generally of small concern and hard to avoid) Often, though, you're either going to be building a bus and applying your own effects and some others (from your OS, from team members, from third party plugins/libraries, etc) or you're going to be writing some kind of effect/generator that gets inserted into somebody else's bus in something like a DAW or game. In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable.
  - spacechild1 2y ago
    Thanks for clarifying. The terms are highly ambigious (see the sibling answer https://news.ycombinator.com/item?id=40930298 https://news.ycombinator.com/item?id=40930298), that's why I asked. Personally, I would rather use the terms "audio pipeline" or "audio graph" instead of the generic "bus". > In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable. Yes. For audio programmers that is obvious, in particular when it comes to plugins, but for novices it might be worth pointing out!
  - RossBencina 2y ago
    > You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In case it is not clear, that is the primary case that is addressed by the linked blog post (source: I wrote the blog post).
    - swatcoder 2y ago
      And likewise: in case it wasn't clear, it's a great article! I wasn't meaning to criticize it, just add a little further perspective for the common scenario that many first-time audio programming folks encounter.
ssfrr 2y ago
Definitely good to keep in mind. The thing that I think is really interesting about audio programming is that you need to be deterministically fast. If your DSP callback executes in 1ms 99.99% of the time but sometimes takes 10ms, you’re hosed. I would love to see a modern take on the real-world risk of various operations that are technically nondeterministic. I wouldn’t be surprised if there are cases where the risk of >1ms latency is like 1e-30, and dogmatically following this advice might be overkill.
- varispeed 2y ago
  The real fun is optimising maths. Remove all divisions. Create LUTs, approximations, CPU specific tricks. Despite the fact CPUs are magnitudes faster now, they are still slow for real time processing.
  - kdjdjjz 2y ago
    Real time does not mean fast, it means deterministic Thus such micro optimizations are seldomly used. Quite the opposite, you try to avoid jitter which could be the result of caches
    - RossBencina 2y ago
      While real-time does not mean fast, micro optimisations are frequently used. No one likes slow DSP audio software.
      - munificent 2y ago
        > No one likes slow DSP audio software. And then there's Diva at its highest output quality setting... :)
        RossBencina 2y ago
        Yes, I did think twice about posting that precisely because of Diva.
RossBencina 2y ago
In the context of the article, I assume that the driver has arranged sufficient buffering so that the jitter in scheduling across a bus (PCI, USB) is masked with respect to the client code. But you are correct that communications overhead can cut into your compute time if it is not addressed. Some audio APIs (e.g. CoreAudio) allow for configuring the buffering margins, so you can trade off buffer latency against available audio compute %. There is a whole world of debate surrounding how to best schedule audio compute (e.g. interrupt driven vs. delay-locked high precision timers). Assuming the context is a desktop OS (which is the context of TFA), I think that the main source of non-determinism is scheduling jitter (the time between the ideal start of your computation, and the time when the OS gives you the CPU to start the computation). Of course if you can't arrange exclusive or max-priority access to a CPU core you're also going to be competing with other processes. Then there is non-deterministic execution time on most modern CPUs due to cache timing effects, superscalar out of order instruction scheduling, inter-core synchronisation, and so on. So yeah, you're going to need some margin unless you're on dedicated hardware with deterministic compute (e.g. a DSP chip).
- swatcoder 2y ago
  No, I'm just talking about the common case where you have some other stuff going on before or after your own audio processing code: a software instrument your framework provides, some AudioUnits or gstreamer nodes adding other effects, the whole device chain in the DAW that's hosting you, etc. All of those things need to get done within your window so you can't use the whole thing for yourself. Most people learning audio programming aren't making a standalone audio app where they do all the processing, or at least not an interesting one. They're usually either making something like a plugin that ends up in somebody else's bus/graph, or something like a game or application that creates a bus/graph and shoves a bunch of different stuff into it.
Derbasti 2y ago
When I was working on desktop audio software, our rule of thumb was to keep CPU occupancy below 30%. More than that, and you'd be sure to get hitching in your audio stream. (This factors in thread contention, other background tasks stealing CPU cycles, file system pauses...) A much different experience from embedded programming, where 99% occupancy is no problem at all.
empiricus 2y ago
Notice that games are often able to render each frame in 5ms: which in practice means run multiple short programs for each pixel you see on the 4k screen. So modern computers are able to do huge huge amount of computation in 5ms (in the order of 10^10 flops, 10^8 bytes). If puny kilobytes of audio data cannot be processed in 5ms it means things are terribly wrong.
- lomase 2y ago
  The GPU is able to paralelizes the drawing of each pixel.
- bluGill 2y ago
  Missing a draw and thus displaying the previous screen for a frame is not noticable in general.
- exDM69 2y ago
  Games do a very impressive amount of work for graphics but there's a huge difference: a dropped/late graphics frame every now and then is not a big deal. An audio glitch is very annoying by comparison, especially if the application is a live musical instrument or something like that. Even the choppy rocket motor sounds of Kerbal Space Program (caused by garbage collector pauses) are infuriating. It's kind of the difference between soft and hard real time systems. Although most audio applications don't strictly qualify as hard real time (missing a deadline is as bad as a total failure) but failing a deadline is much worse than in graphics.
  - izacus 2y ago
    Frame pacing issues due to dropped frames are absolutely a huge deal in games.
    - exDM69 2y ago
      Yes, but audio glitches are much worse for the experience.
jcelerier 2y ago
Depends on where you run your audio. I've worked on embedded devices where we could just use isolcpu and IRQ controls to make sure specific cores were only ever used for the audio thread of the app.

spacechild1 2y ago

What do you mean by "bus" and "module" in this context?

GrantMoyer 2y ago
A module is a piece of software or hardware which is independent in some way. A bus is a shared medium of communication[1]. Often, busses are time-division multiplexed[2], so if you want to use the bus, but another module is already using it, you need to wait. For example, if your audio buffers are ultimately submitted to a sound card over a PCI bus, the submission may need to wait for any ongoing transactions on the PCI bus, such as messages to a graphics card. [1]: https://en.wikipedia.org/wiki/Bus_(computing) https://en.wikipedia.org/wiki/Bus_(computing) [2]: https://en.wikipedia.org/wiki/Time-division_multiplexing https://en.wikipedia.org/wiki/Time-division_multiplexing
- spacechild1 2y ago
  That is one possible interpretation, but not what they meant. That's why I asked because I wasn't sure :)
swatcoder 2y ago
"Bus" (as I was using it) is the path from some audio source to some audio destination and a "module" (as used) would be something that takes a buffer of samples on that bus and does something with it. You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In this case, you'd more or less control the whole bus and all the code processing samples on it and have a fairly true sense of your deadline. (The OS and drivers would still be introducing some overhead for mixing or resampling, etc, but that's generally of small concern and hard to avoid) Often, though, you're either going to be building a bus and applying your own effects and some others (from your OS, from team members, from third party plugins/libraries, etc) or you're going to be writing some kind of effect/generator that gets inserted into somebody else's bus in something like a DAW or game. In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable.
- spacechild1 2y ago
  Thanks for clarifying. The terms are highly ambigious (see the sibling answer https://news.ycombinator.com/item?id=40930298 https://news.ycombinator.com/item?id=40930298), that's why I asked. Personally, I would rather use the terms "audio pipeline" or "audio graph" instead of the generic "bus". > In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable. Yes. For audio programmers that is obvious, in particular when it comes to plugins, but for novices it might be worth pointing out!
- RossBencina 2y ago
  > You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In case it is not clear, that is the primary case that is addressed by the linked blog post (source: I wrote the blog post).
  - swatcoder 2y ago
    And likewise: in case it wasn't clear, it's a great article! I wasn't meaning to criticize it, just add a little further perspective for the common scenario that many first-time audio programming folks encounter.

ssfrr 2y ago

Definitely good to keep in mind. The thing that I think is really interesting about audio programming is that you need to be deterministically fast. If your DSP callback executes in 1ms 99.99% of the time but sometimes takes 10ms, you’re hosed. I would love to see a modern take on the real-world risk of various operations that are technically nondeterministic. I wouldn’t be surprised if there are cases where the risk of >1ms latency is like 1e-30, and dogmatically following this advice might be overkill.

varispeed 2y ago
The real fun is optimising maths. Remove all divisions. Create LUTs, approximations, CPU specific tricks. Despite the fact CPUs are magnitudes faster now, they are still slow for real time processing.
- kdjdjjz 2y ago
  Real time does not mean fast, it means deterministic Thus such micro optimizations are seldomly used. Quite the opposite, you try to avoid jitter which could be the result of caches
  - RossBencina 2y ago
    While real-time does not mean fast, micro optimisations are frequently used. No one likes slow DSP audio software.
    - munificent 2y ago
      > No one likes slow DSP audio software. And then there's Diva at its highest output quality setting... :)
      - RossBencina 2y ago
        Yes, I did think twice about posting that precisely because of Diva.

RossBencina 2y ago

In the context of the article, I assume that the driver has arranged sufficient buffering so that the jitter in scheduling across a bus (PCI, USB) is masked with respect to the client code. But you are correct that communications overhead can cut into your compute time if it is not addressed. Some audio APIs (e.g. CoreAudio) allow for configuring the buffering margins, so you can trade off buffer latency against available audio compute %. There is a whole world of debate surrounding how to best schedule audio compute (e.g. interrupt driven vs. delay-locked high precision timers). Assuming the context is a desktop OS (which is the context of TFA), I think that the main source of non-determinism is scheduling jitter (the time between the ideal start of your computation, and the time when the OS gives you the CPU to start the computation). Of course if you can't arrange exclusive or max-priority access to a CPU core you're also going to be competing with other processes. Then there is non-deterministic execution time on most modern CPUs due to cache timing effects, superscalar out of order instruction scheduling, inter-core synchronisation, and so on. So yeah, you're going to need some margin unless you're on dedicated hardware with deterministic compute (e.g. a DSP chip).

swatcoder 2y ago
No, I'm just talking about the common case where you have some other stuff going on before or after your own audio processing code: a software instrument your framework provides, some AudioUnits or gstreamer nodes adding other effects, the whole device chain in the DAW that's hosting you, etc. All of those things need to get done within your window so you can't use the whole thing for yourself. Most people learning audio programming aren't making a standalone audio app where they do all the processing, or at least not an interesting one. They're usually either making something like a plugin that ends up in somebody else's bus/graph, or something like a game or application that creates a bus/graph and shoves a bunch of different stuff into it.

Derbasti 2y ago

When I was working on desktop audio software, our rule of thumb was to keep CPU occupancy below 30%. More than that, and you'd be sure to get hitching in your audio stream. (This factors in thread contention, other background tasks stealing CPU cycles, file system pauses...) A much different experience from embedded programming, where 99% occupancy is no problem at all.

empiricus 2y ago

Notice that games are often able to render each frame in 5ms: which in practice means run multiple short programs for each pixel you see on the 4k screen. So modern computers are able to do huge huge amount of computation in 5ms (in the order of 10^10 flops, 10^8 bytes). If puny kilobytes of audio data cannot be processed in 5ms it means things are terribly wrong.

lomase 2y ago
The GPU is able to paralelizes the drawing of each pixel.
bluGill 2y ago
Missing a draw and thus displaying the previous screen for a frame is not noticable in general.
exDM69 2y ago
Games do a very impressive amount of work for graphics but there's a huge difference: a dropped/late graphics frame every now and then is not a big deal. An audio glitch is very annoying by comparison, especially if the application is a live musical instrument or something like that. Even the choppy rocket motor sounds of Kerbal Space Program (caused by garbage collector pauses) are infuriating. It's kind of the difference between soft and hard real time systems. Although most audio applications don't strictly qualify as hard real time (missing a deadline is as bad as a total failure) but failing a deadline is much worse than in graphics.
- izacus 2y ago
  Frame pacing issues due to dropped frames are absolutely a huge deal in games.
  - exDM69 2y ago
    Yes, but audio glitches are much worse for the experience.

jcelerier 2y ago

Depends on where you run your audio. I've worked on embedded devices where we could just use isolcpu and IRQ controls to make sure specific cores were only ever used for the audio thread of the app.