10 ms·

Definitely good to keep in mind. The thing that I think is really interesting about audio programming is that you need to be deterministically fast. If your DSP

by ssfrr 2y ago

Definitely good to keep in mind. The thing that I think is really interesting about audio programming is that you need to be deterministically fast. If your DSP callback executes in 1ms 99.99% of the time but sometimes takes 10ms, you’re hosed. I would love to see a modern take on the real-world risk of various operations that are technically nondeterministic. I wouldn’t be surprised if there are cases where the risk of >1ms latency is like 1e-30, and dogmatically following this advice might be overkill.

varispeed 2y ago
The real fun is optimising maths. Remove all divisions. Create LUTs, approximations, CPU specific tricks. Despite the fact CPUs are magnitudes faster now, they are still slow for real time processing.
- kdjdjjz 2y ago
  Real time does not mean fast, it means deterministic Thus such micro optimizations are seldomly used. Quite the opposite, you try to avoid jitter which could be the result of caches
  - RossBencina 2y ago
    While real-time does not mean fast, micro optimisations are frequently used. No one likes slow DSP audio software.
    - munificent 2y ago
      > No one likes slow DSP audio software. And then there's Diva at its highest output quality setting... :)
      - RossBencina 2y ago
        Yes, I did think twice about posting that precisely because of Diva.
  - PaulDavisThe1st 2y ago
    Jitter does not matter if deadlines are met. It only matters if it can cause deadlines to be missed (sometimes).
    - nsguy 2y ago
      If you have a buffer that's being clocked out and your goal is to keep data flowing, the jitter is going to influence how small your buffer can be. Let's say you're producing 56Khz audio, the best you can do is produce a [sample] exactly at that frequency. If you have 1ms jitter now you need a 1ms buffer so you have delay. If jitter is small enough, like 0.1ns jitter in some SIMD calculation, then for all intent and purpose it doesn't matter for an audio application...
      - PaulDavisThe1st 2y ago
        You've just restated my point. If the deadlines are met, jitter doesn't matter. Ergo, you can't meet deadlines if your jitter is too large. Otherwise, it doesn't matter.
        varispeed 2y ago
        Basically "It doesn't matter when it doesn't matter".
        nsguy 2y ago
        Wouldn't the deadline be now+zero for real time audio applications? If I'm building a guitar pedal (random example) ideally I want no delay from the input to the output. Any digital delay makes things strictly worse and so any jitter matters. That said, the difference between zero and very close to zero does become a moot point given small enough values for any practical purpose.
        PaulDavisThe1st 2y ago
        There are some digital audio systems that do sample-by-sample processing. Old school digidesign, for example. But very little digital audio gear works that way these days. The buffer sizes may be small (e.g 8 or 16 samples), but most hardware uses block structured (buffer by buffer) processing. So there's always a delay, even if 1 sample.
- PaulDavisThe1st 2y ago
  > Create LUTs This has been slower for most things that raw computation for well over a decade (probably more like two).
  - kaba0 2y ago
    Yeah, isn’t hitting memory (especially if it can’t fit in L1-2 cache) one of the biggest sources of latency? Especially that on modern CPUs it is almost impossible to max out the arithmetic units, outside of microbenchmarks?
  - lomase 2y ago
    Not at all, when you work with DSP even nowdays using LUTs is very common and usually faster. You are not saving a sin table, but very complex differential equations.
  - varispeed 2y ago
    If there are complex equations involved, it absolutely is faster. You can also create intermediate LUTs, so the tables are small and fit in cache and then do interpolation on the fly.
- nsguy 2y ago
  You don't really do these any more on a modern CPU. This is stuff I used to do 30 years ago and you might still do if you're on a micro-controller or some other tiny system. The CPUs aren't slow. Tne main problem is if the OS doesn't schedule your process it doesn't matter how fast the CPU is.
- Derbasti 2y ago
  This is great fun! But it's much more prevalent in embedded DSP than desktop.
rzzzt 2y ago
Use of Ethernet in real-time systems. Packet loss, collision rate, jitter is """good enough""" so it became an acceptable replacement of eg. ATM.
- lukeh 2y ago
  Or you use AVB/TSN which gives you stronger guarantees, but requires cooperation of all bridges (switches).
- nsguy 2y ago
  Yes. Most modern Ethernet isn't running on shared media (i.e. there are no collisions) and for the most part no packet loss as long as there's no congestion. For networks and for the CPU, when you're fast enough the jitter matters less, if the cpu or the network "takes a break" (from the application perspective), it tends to be a very short break on really fast networks or cpus. e.g. if a packet gets in front of you in 10Mbps Ethernet that's a big deal for an audio application but a packet ahead of you in 10Gbps Ethernet isn't much of a delay for audio. 1ms vs. 1us sort of thing. [fixed typo]
RossBencina 2y ago
> deterministically fast Indeed, like all real-time systems you need to think in terms of worst-case time complexity, not amortized complexity.
jancsika 2y ago
> If your DSP callback executes in 1ms 99.99% of the time but sometimes takes 10ms, you’re hosed. I tend to agree, but... From my recollection of using Zoom-- it has this bizarre but workable recovery method for network interruptions. Either the server or the client keeps some amount of the last input audio in a buffer. Then if the server detects connection problems at time 't', it grabs the buffer from t - 1 seconds all the way until the server detects better connectivity. Then it starts a race condition, playing back that amount of the buffer to all clients at something like 1.5 speed. From what I remember, this algo typically wins the race and saves the client from having to repeat themselves. That's not happening inside a DSP routine. But my point is that some clever engineer(s) at Zoom realized that missing deadlines in audio delivery does not necessarily mean "hosed." I'm also going to rankly speculate that every other video conferencing tool hard-coupled missing deadlines with "hosed," and that's why Zoom is the only one where I've ever experienced the benefit of that feature.
- ssfrr 2y ago
  The context for this article is writing pro audio software, where that kind of distortion would generally be as bad as a dropout, if not worse.
  - wizardforhire 2y ago
    Yeah, 5ms is the threshold for noticeability as far as latency in pro-audio. Its like frame-rate for pro-gamers. The problem is your target user is highly specialized out side the the norms by a large margin. What makes audio even more difficult is that sub ms issues can cause phase and frequency distortion that can become even more noticeable than latency alone.
    - PaulDavisThe1st 2y ago
      1. you do not need to be a highly specialized target user to detect latency between pressing a key on a MIDI keyboard and the corresponding sound being produced. 2. 3ms is typical in-air latency between a typical DAW user and their near-field monitors, so claims about sensitivity to times much lower than 5msec should be taken with some skepticism 3. In live contexts, many drum + bass pairings have more than 10ms of air latency between them, so ditto #2 4. On the other hand, no good reason to add to latency 5. For performance purposes, jitter is much worse than latency. Pipe organ players rapidly learn to deal with even whole seconds of latency, but almost nobody can deal with jitter (essentially, variable, unpredictable latency) 6. There are no sub-ms issues that will cause phase and frequency distortion. Those come from DSP errors, not handling of latency, which is just about always a constant, fixed feature of the data signal path. You may be thinking of stuff like comb filtering, but this is not related to the latency in the signal path in a correct setup.
RossBencina 2y ago
> dogmatically following this advice might be overkill It depends on your appetite for risk and the cost of failure. A big part of the problem is that general purpose computing systems (operating systems and hardware) are not engineered as real-time systems and there are rarely vendor guarantees with respect to real-time behavior. Under such circumstances, my position is that you need to code defensively. For example, if your operating system memory allocator does not guarantee a worst-case bound on execution time, do not use it in a real-time context.
- ssfrr 2y ago
  I don't mean to devalue the advice here. I think it's spot on, and I unreservedly recommend this article to folks who want to learn about writing reliable audio software. I think in essence I'm repeating the comments of Justin from Cockos, which you summarize [1]: > It is basically saying that you can reduce the risk of priority inversion to the point where the probability is too low to worry about. In that comment you also say: > 100% certainty can’t be guaranteed without a hard real-time OS. However 5ms is now considered a relatively high latency setting in pro/prosumer audio circles Which I interpret as acknowledging that we're already forced into the regime of establishing an acceptable level of risk. My point is that I would love to see more data on the actual latency distributions we can expect, so that we can make more informed risk assessments. For example, I know that not all `std::atomic` operations are lock-free, but when the critical section is so small, is it really a problem in practice? I want histograms! [1]: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing#comment-87 http://www.rossbencina.com/code/real-time-audio-programming-...
munificent 2y ago
I don't know. When it comes to real-time audio... imagine a huge festival with a giant wall of speakers blasting at the audience. If the audio playback glitches and you something like a 22kHz buzz (alternating two samples), that is a lot of fried ears.
- wizardforhire 2y ago
  This scenario is the stuff of nightmares for me! When you have 100k people paying $500 to the sky is the limit, failure is not an option. Increasingly audio engineers and subsequently performers are at the mercy of the latest jr developers who don’t have to live with the failures of their short sightedness. Grimes’ Coachella set case in point. Wholly due to pioneer ignoring their users for over a decade. Sometimes we don’t have 3 days to copy files to a usb drive but I digress.
  - PaulDavisThe1st 2y ago
    Apparently, failure was an option. Just not a very popular one.
    - fxtentacle 2y ago
      Grimes failure was still pleasant when you compare it to the mayhem you get when the DSP inside the amplifier system glitches for a few samples. What do you think happens a dense crowd of 500+ people suddenly starts to have excruciating ear pain?
      - jalk 2y ago
        Is this something that happens often or are you simply speculating?
  - alanbernstein 2y ago
    Heh, I thought it was odd you referenced a ten year old show, but I guess she made a similar mistake twice. Her 2014 Coachella set was a total mess.

by ssfrr 2y ago

varispeed 2y ago

The real fun is optimising maths. Remove all divisions. Create LUTs, approximations, CPU specific tricks. Despite the fact CPUs are magnitudes faster now, they are still slow for real time processing.

kdjdjjz 2y ago
Real time does not mean fast, it means deterministic Thus such micro optimizations are seldomly used. Quite the opposite, you try to avoid jitter which could be the result of caches
- RossBencina 2y ago
  While real-time does not mean fast, micro optimisations are frequently used. No one likes slow DSP audio software.
  - munificent 2y ago
    > No one likes slow DSP audio software. And then there's Diva at its highest output quality setting... :)
    - RossBencina 2y ago
      Yes, I did think twice about posting that precisely because of Diva.
- PaulDavisThe1st 2y ago
  Jitter does not matter if deadlines are met. It only matters if it can cause deadlines to be missed (sometimes).
  - nsguy 2y ago
    If you have a buffer that's being clocked out and your goal is to keep data flowing, the jitter is going to influence how small your buffer can be. Let's say you're producing 56Khz audio, the best you can do is produce a [sample] exactly at that frequency. If you have 1ms jitter now you need a 1ms buffer so you have delay. If jitter is small enough, like 0.1ns jitter in some SIMD calculation, then for all intent and purpose it doesn't matter for an audio application...
    - PaulDavisThe1st 2y ago
      You've just restated my point. If the deadlines are met, jitter doesn't matter. Ergo, you can't meet deadlines if your jitter is too large. Otherwise, it doesn't matter.
      - varispeed 2y ago
        Basically "It doesn't matter when it doesn't matter".
      - nsguy 2y ago
        Wouldn't the deadline be now+zero for real time audio applications? If I'm building a guitar pedal (random example) ideally I want no delay from the input to the output. Any digital delay makes things strictly worse and so any jitter matters. That said, the difference between zero and very close to zero does become a moot point given small enough values for any practical purpose.
        PaulDavisThe1st 2y ago
        There are some digital audio systems that do sample-by-sample processing. Old school digidesign, for example. But very little digital audio gear works that way these days. The buffer sizes may be small (e.g 8 or 16 samples), but most hardware uses block structured (buffer by buffer) processing. So there's always a delay, even if 1 sample.
PaulDavisThe1st 2y ago
> Create LUTs This has been slower for most things that raw computation for well over a decade (probably more like two).
- kaba0 2y ago
  Yeah, isn’t hitting memory (especially if it can’t fit in L1-2 cache) one of the biggest sources of latency? Especially that on modern CPUs it is almost impossible to max out the arithmetic units, outside of microbenchmarks?
- lomase 2y ago
  Not at all, when you work with DSP even nowdays using LUTs is very common and usually faster. You are not saving a sin table, but very complex differential equations.
- varispeed 2y ago
  If there are complex equations involved, it absolutely is faster. You can also create intermediate LUTs, so the tables are small and fit in cache and then do interpolation on the fly.
nsguy 2y ago
You don't really do these any more on a modern CPU. This is stuff I used to do 30 years ago and you might still do if you're on a micro-controller or some other tiny system. The CPUs aren't slow. Tne main problem is if the OS doesn't schedule your process it doesn't matter how fast the CPU is.
Derbasti 2y ago
This is great fun! But it's much more prevalent in embedded DSP than desktop.

rzzzt 2y ago

Use of Ethernet in real-time systems. Packet loss, collision rate, jitter is """good enough""" so it became an acceptable replacement of eg. ATM.

lukeh 2y ago
Or you use AVB/TSN which gives you stronger guarantees, but requires cooperation of all bridges (switches).
nsguy 2y ago
Yes. Most modern Ethernet isn't running on shared media (i.e. there are no collisions) and for the most part no packet loss as long as there's no congestion. For networks and for the CPU, when you're fast enough the jitter matters less, if the cpu or the network "takes a break" (from the application perspective), it tends to be a very short break on really fast networks or cpus. e.g. if a packet gets in front of you in 10Mbps Ethernet that's a big deal for an audio application but a packet ahead of you in 10Gbps Ethernet isn't much of a delay for audio. 1ms vs. 1us sort of thing. [fixed typo]

RossBencina 2y ago

> deterministically fast Indeed, like all real-time systems you need to think in terms of worst-case time complexity, not amortized complexity.

jancsika 2y ago

> If your DSP callback executes in 1ms 99.99% of the time but sometimes takes 10ms, you’re hosed. I tend to agree, but... From my recollection of using Zoom-- it has this bizarre but workable recovery method for network interruptions. Either the server or the client keeps some amount of the last input audio in a buffer. Then if the server detects connection problems at time 't', it grabs the buffer from t - 1 seconds all the way until the server detects better connectivity. Then it starts a race condition, playing back that amount of the buffer to all clients at something like 1.5 speed. From what I remember, this algo typically wins the race and saves the client from having to repeat themselves. That's not happening inside a DSP routine. But my point is that some clever engineer(s) at Zoom realized that missing deadlines in audio delivery does not necessarily mean "hosed." I'm also going to rankly speculate that every other video conferencing tool hard-coupled missing deadlines with "hosed," and that's why Zoom is the only one where I've ever experienced the benefit of that feature.

ssfrr 2y ago
The context for this article is writing pro audio software, where that kind of distortion would generally be as bad as a dropout, if not worse.
- wizardforhire 2y ago
  Yeah, 5ms is the threshold for noticeability as far as latency in pro-audio. Its like frame-rate for pro-gamers. The problem is your target user is highly specialized out side the the norms by a large margin. What makes audio even more difficult is that sub ms issues can cause phase and frequency distortion that can become even more noticeable than latency alone.
  - PaulDavisThe1st 2y ago
    1. you do not need to be a highly specialized target user to detect latency between pressing a key on a MIDI keyboard and the corresponding sound being produced. 2. 3ms is typical in-air latency between a typical DAW user and their near-field monitors, so claims about sensitivity to times much lower than 5msec should be taken with some skepticism 3. In live contexts, many drum + bass pairings have more than 10ms of air latency between them, so ditto #2 4. On the other hand, no good reason to add to latency 5. For performance purposes, jitter is much worse than latency. Pipe organ players rapidly learn to deal with even whole seconds of latency, but almost nobody can deal with jitter (essentially, variable, unpredictable latency) 6. There are no sub-ms issues that will cause phase and frequency distortion. Those come from DSP errors, not handling of latency, which is just about always a constant, fixed feature of the data signal path. You may be thinking of stuff like comb filtering, but this is not related to the latency in the signal path in a correct setup.

RossBencina 2y ago

> dogmatically following this advice might be overkill It depends on your appetite for risk and the cost of failure. A big part of the problem is that general purpose computing systems (operating systems and hardware) are not engineered as real-time systems and there are rarely vendor guarantees with respect to real-time behavior. Under such circumstances, my position is that you need to code defensively. For example, if your operating system memory allocator does not guarantee a worst-case bound on execution time, do not use it in a real-time context.

ssfrr 2y ago
I don't mean to devalue the advice here. I think it's spot on, and I unreservedly recommend this article to folks who want to learn about writing reliable audio software. I think in essence I'm repeating the comments of Justin from Cockos, which you summarize [1]: > It is basically saying that you can reduce the risk of priority inversion to the point where the probability is too low to worry about. In that comment you also say: > 100% certainty can’t be guaranteed without a hard real-time OS. However 5ms is now considered a relatively high latency setting in pro/prosumer audio circles Which I interpret as acknowledging that we're already forced into the regime of establishing an acceptable level of risk. My point is that I would love to see more data on the actual latency distributions we can expect, so that we can make more informed risk assessments. For example, I know that not all `std::atomic` operations are lock-free, but when the critical section is so small, is it really a problem in practice? I want histograms! [1]: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing#comment-87 http://www.rossbencina.com/code/real-time-audio-programming-...

munificent 2y ago

I don't know. When it comes to real-time audio... imagine a huge festival with a giant wall of speakers blasting at the audience. If the audio playback glitches and you something like a 22kHz buzz (alternating two samples), that is a lot of fried ears.

wizardforhire 2y ago
This scenario is the stuff of nightmares for me! When you have 100k people paying $500 to the sky is the limit, failure is not an option. Increasingly audio engineers and subsequently performers are at the mercy of the latest jr developers who don’t have to live with the failures of their short sightedness. Grimes’ Coachella set case in point. Wholly due to pioneer ignoring their users for over a decade. Sometimes we don’t have 3 days to copy files to a usb drive but I digress.
- PaulDavisThe1st 2y ago
  Apparently, failure was an option. Just not a very popular one.
  - fxtentacle 2y ago
    Grimes failure was still pleasant when you compare it to the mayhem you get when the DSP inside the amplifier system glitches for a few samples. What do you think happens a dense crowd of 500+ people suddenly starts to have excruciating ear pain?
    - jalk 2y ago
      Is this something that happens often or are you simply speculating?
- alanbernstein 2y ago
  Heh, I thought it was odd you referenced a ten year old show, but I guess she made a similar mistake twice. Her 2014 Coachella set was a total mess.