9 ms·
Real-time audio programming 101: time waits for nothing (2011)
- deleted 2y ago[deleted]
- swatcoder 2y ago(2011) But a great summary and mostly evergreen One practical reality it doesn't share is that your audio processing (or generation) code is often going to be running in a bus shared by a ton of other modules and so you don't have the luxury of using "5.6ms" as your deadline for a 5.6ms buffer. Your responsibility, often, is to just get as performant as reasonably possible so that everything on the bus can be processed in those 5.6ms. The pressure is usually much higher than the buffer length suggests.
- spacechild1 2y agoWhat do you mean by "bus" and "module" in this context?
- GrantMoyer 2y agoA module is a piece of software or hardware which is independent in some way. A bus is a shared medium of communication[1]. Often, busses are time-division multiplexed[2], so if you want to use the bus, but another module is already using it, you need to wait. For example, if your audio buffers are ultimately submitted to a sound card over a PCI bus, the submission may need to wait for any ongoing transactions on the PCI bus, such as messages to a graphics card. [1]: https://en.wikipedia.org/wiki/Bus_(computing) https://en.wikipedia.org/wiki/Bus_(computing) [2]: https://en.wikipedia.org/wiki/Time-division_multiplexing https://en.wikipedia.org/wiki/Time-division_multiplexing
- spacechild1 2y agoThat is one possible interpretation, but not what they meant. That's why I asked because I wasn't sure :)
- swatcoder 2y ago"Bus" (as I was using it) is the path from some audio source to some audio destination and a "module" (as used) would be something that takes a buffer of samples on that bus and does something with it. You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In this case, you'd more or less control the whole bus and all the code processing samples on it and have a fairly true sense of your deadline. (The OS and drivers would still be introducing some overhead for mixing or resampling, etc, but that's generally of small concern and hard to avoid) Often, though, you're either going to be building a bus and applying your own effects and some others (from your OS, from team members, from third party plugins/libraries, etc) or you're going to be writing some kind of effect/generator that gets inserted into somebody else's bus in something like a DAW or game. In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable.
- spacechild1 2y agoThanks for clarifying. The terms are highly ambigious (see the sibling answer https://news.ycombinator.com/item?id=40930298 https://news.ycombinator.com/item?id=40930298), that's why I asked. Personally, I would rather use the terms "audio pipeline" or "audio graph" instead of the generic "bus". > In all these cases, you need to assume that all processing code that isn't yours needs all the time that you can leave for it and just make your own code as efficient as is reasonable. Yes. For audio programmers that is obvious, in particular when it comes to plugins, but for novices it might be worth pointing out!
- RossBencina 2y ago> You might sometimes build an app where (through your operating system) you connect directly with an input device and/or output device and then do all the audio processing yourself. In case it is not clear, that is the primary case that is addressed by the linked blog post (source: I wrote the blog post).
- swatcoder 2y ago
- ssfrr 2y agoDefinitely good to keep in mind. The thing that I think is really interesting about audio programming is that you need to be deterministically fast. If your DSP callback executes in 1ms 99.99% of the time but sometimes takes 10ms, you’re hosed. I would love to see a modern take on the real-world risk of various operations that are technically nondeterministic. I wouldn’t be surprised if there are cases where the risk of >1ms latency is like 1e-30, and dogmatically following this advice might be overkill.
- varispeed 2y agoThe real fun is optimising maths. Remove all divisions. Create LUTs, approximations, CPU specific tricks. Despite the fact CPUs are magnitudes faster now, they are still slow for real time processing.
- kdjdjjz 2y agoReal time does not mean fast, it means deterministic Thus such micro optimizations are seldomly used. Quite the opposite, you try to avoid jitter which could be the result of caches
- RossBencina 2y agoWhile real-time does not mean fast, micro optimisations are frequently used. No one likes slow DSP audio software.
- munificent 2y ago> No one likes slow DSP audio software. And then there's Diva at its highest output quality setting... :)
- RossBencina 2y agoYes, I did think twice about posting that precisely because of Diva.
- RossBencina 2y agoIn the context of the article, I assume that the driver has arranged sufficient buffering so that the jitter in scheduling across a bus (PCI, USB) is masked with respect to the client code. But you are correct that communications overhead can cut into your compute time if it is not addressed. Some audio APIs (e.g. CoreAudio) allow for configuring the buffering margins, so you can trade off buffer latency against available audio compute %. There is a whole world of debate surrounding how to best schedule audio compute (e.g. interrupt driven vs. delay-locked high precision timers). Assuming the context is a desktop OS (which is the context of TFA), I think that the main source of non-determinism is scheduling jitter (the time between the ideal start of your computation, and the time when the OS gives you the CPU to start the computation). Of course if you can't arrange exclusive or max-priority access to a CPU core you're also going to be competing with other processes. Then there is non-deterministic execution time on most modern CPUs due to cache timing effects, superscalar out of order instruction scheduling, inter-core synchronisation, and so on. So yeah, you're going to need some margin unless you're on dedicated hardware with deterministic compute (e.g. a DSP chip).
- swatcoder 2y agoNo, I'm just talking about the common case where you have some other stuff going on before or after your own audio processing code: a software instrument your framework provides, some AudioUnits or gstreamer nodes adding other effects, the whole device chain in the DAW that's hosting you, etc. All of those things need to get done within your window so you can't use the whole thing for yourself. Most people learning audio programming aren't making a standalone audio app where they do all the processing, or at least not an interesting one. They're usually either making something like a plugin that ends up in somebody else's bus/graph, or something like a game or application that creates a bus/graph and shoves a bunch of different stuff into it.
- Derbasti 2y agoWhen I was working on desktop audio software, our rule of thumb was to keep CPU occupancy below 30%. More than that, and you'd be sure to get hitching in your audio stream. (This factors in thread contention, other background tasks stealing CPU cycles, file system pauses...) A much different experience from embedded programming, where 99% occupancy is no problem at all.
- empiricus 2y agoNotice that games are often able to render each frame in 5ms: which in practice means run multiple short programs for each pixel you see on the 4k screen. So modern computers are able to do huge huge amount of computation in 5ms (in the order of 10^10 flops, 10^8 bytes). If puny kilobytes of audio data cannot be processed in 5ms it means things are terribly wrong.
- lomase 2y agoThe GPU is able to paralelizes the drawing of each pixel.
- bluGill 2y agoMissing a draw and thus displaying the previous screen for a frame is not noticable in general.
- exDM69 2y agoGames do a very impressive amount of work for graphics but there's a huge difference: a dropped/late graphics frame every now and then is not a big deal. An audio glitch is very annoying by comparison, especially if the application is a live musical instrument or something like that. Even the choppy rocket motor sounds of Kerbal Space Program (caused by garbage collector pauses) are infuriating. It's kind of the difference between soft and hard real time systems. Although most audio applications don't strictly qualify as hard real time (missing a deadline is as bad as a total failure) but failing a deadline is much worse than in graphics.
- jcelerier 2y agoDepends on where you run your audio. I've worked on embedded devices where we could just use isolcpu and IRQ controls to make sure specific cores were only ever used for the audio thread of the app.
- user_7832 2y agoSlightly tangential, does anyone know any good (windows based) DSP software? EquilizerAPO is decent in theory but beyond being clunky to use unfortunately doesn't even seem to work 90% of the time.
- Ylpertnodi 2y agowww.airwindows.com
- bratwurst3000 2y agoI think camilla dsp works for windows.
- chresko 2y agoSuperCollider
- spacechild1 2y agoGraphical: Pure Data, Max/MSP Text based: SuperCollider, Csound, Chuck
- Ylpertnodi 2y agohttps://www.airwindows.com/consolidated/ https://www.airwindows.com/consolidated/
- RossBencina 2y agoAudioMulch?
- Optimal_Persona 2y agoI wish you were still actively developing it, I haven't been able to run it on Mac for a while now. More than any other recording software, AudioMulch gave me a taste for digital audio and specifically realtime resampling. Since using AudioMulch, any DAW that doesn't let me record arbitrary combos of audio streams in realtime feels hopelessly limited.
- anigbrowl 2y ago
- spacechild1 2y agoA timeless classic! This is the first thing I always recommend to anyone interested in real-time audio programming.
- chaosprint 2y agoGreat resource! For those interested in learning the fundamentals of audio programming, I highly recommend starting with Rust. the cpal library in Rust is excellent for developing cross-platform desktop applications. I'm currently maintaining this library: https://github.com/chaosprint/asak https://github.com/chaosprint/asak It's a cross-platform audio recording/playback CLI tool with TUI. The source code is very simple to read. PRs are welcomed and I really hope Linux users can help to test and review new PRs :) When developing Glicol(https://glicol.org https://glicol.org), I documented my experience of "fighting" with real-time audio in the browser in this paper: https://webaudioconf.com/_data/papers/pdf/2021/2021_8.pdf https://webaudioconf.com/_data/papers/pdf/2021/2021_8.pdf Throughout the process, Paul Adenot's work was immensely helpful. I highly recommend his blog: https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ https://blog.paul.cx/post/profiling-firefox-real-time-media-... I am currently writing a wasm audio module system, and hope to publish it here soon.
- nyanpasu64 2y agoIs it still the case that cpal doesn't support "synchronous" duplex audio where the program inputs audio from a source and outputs it to a sink (either with feedback or outputting unrelated audio), with an integer number of periods (as little as 2) of software-level latency if you copy source buffers to the sink? Last time I used it, each stream is opened in input or output mode and opening both does not run with any guaranteed timing relation.
- dmix 2y agoIs there a good "toolbox" style cli for audio? Like pitch shifting and time stretching etc
- jim-jim-jim 2y agosox
- smj-edison 2y agoI have mixed feelings on cpal: on the one hand, it's been really wonderful to have a library that just works on different platforms. On the other hand, it's an absolute pain in the butt for doing anything simple. I really wish it would have a simple interface for when I'm only worried about floating point data (I ended up creating my own library to wrap cpal's idiosyncrasies for my mixed midi/audio node program: https://github.com/smj-edison/clocked https://github.com/smj-edison/clocked).
- marcod 2y agoOff topic. Anybody else like Thursday Next? Had to think of "Time waits for no man!"
- brcmthrowaway 2y agoThis seems super outdated. Isn't CoreAudio HW accelerated now?
- raphlinus 2y agoIt continues to be relevant. On modern computers, mobile especially, there are are more and more things that can glitch audio from being actually real-time. Probably one of the worst offenders is power management; it is extremely likely that the power management governor has no awareness of upcoming audio deadlines. Obviously the happy case is when all the audio processing is done in a DSP where scheduling is deterministic, but it's rare to be able to count on that. Part of the problem is that modern computers are so fast that people expect them to handle audio tasks without breathing hard. But that speed is usually measured as throughput rather than worst-case latency. The advice I'd give to anybody building audio today is to relentlessly measure all potential sources of scheduling jitter end-to-end. Once you know that, it becomes clearer how to address it.
- nyanpasu64 2y agoHow do you measure scheduling jitter of audio drivers and userspace? I hear real-time priority or kernels improve latency? (There was some controversy about Ubuntu Studio or something switching to RT kernels with more missed deadlines than RT-PREEMPT, but I don't know how to quantify this stuff.)
- binary132 2y agoalso interested in the answer to this
- raphlinus 2y agoThat's actually a very good question. To some extent, performance analysis tools may be able to give you high-accuracy timestamps of things involving context switches and other things that can cause scheduling jitter. If you can get access to things like the fill level of FIFO buffers, even better. You may also be able to do experiments like cutting buffer sizes down to the bone to see how low they can go without glitching. Of course, it's not unusual that the many layers of abstraction in modern systems actively frustrate getting real performance data. But dealing with that is part of the requirements of doing real engineering.
- jmkr 2y agoAs a web developer, learning music and audio programming makes my mind melt. We often say "real time" when we mean "fast." But in audio real time means "really fast, all the time" and somewhat deterministically. If your tempo drifts, then you're not going to hear the rhythm correctly. If you have a bit of latency on your instrument, it's like turning on a delay pedal where the only signal coming through is the delay. One might assume if you just follow audio programming guides then you can do all this, but you still need to have your system setup to handle real time audio, in addition to your program. It's all noticeable.
- hrkfmud50k 2y agoif you think that's cool then you may also like a "hard" real time programs e.g. ABS brakes https://en.wikipedia.org/wiki/Real-time_computing#Criteria_for_real-time_computing https://en.wikipedia.org/wiki/Real-time_computing#Criteria_f...
- duped 2y ago> We often say "real time" when we mean "fast." But in audio real time means "really fast, all the time" and somewhat deterministically. As a former developer of real time software, the usage of "real time" to mean "fast" makes me cringe a bit whenever I read it. If there's a TCP/IP stack in the middle of something, it's probably not "real time." "real time" means there's a deadline. Soft real time means missing the deadline is a problem, possibly a bug, and quite bad. Hard real time means the "dead" part of "deadline" could be literal, either in terms of your program (a missed deadline is an irrecoverable error) or the humans that need the program to make the deadline are no longer alive.
- jmkr 2y agoWhat is it like working with hard real time? As in, is there some tooling to determine how long a given function will take to run to meet a deadline? It starts to sound like an NP problem, but we have other kinds of correctness and proof testing. I suppose it's hard to make guarantees with different environments and hardware, but I realized when we (non-realtime people) ship software we don't really have guarantees for when our functions run.
- white_beach 2y agoi thought glitch music a good
- demondemidi 2y agoIf you’re worried about glitches during live performances there’s a fool proof solution: play real instruments. ;)
- PaulDavisThe1st 2y ago... and if the FOH engineer is using a poorly engineered digital console, or even a DAW, for mixing ... ?
- shiroiushi 2y agoThat's great if you don't need any amplification at all, or perhaps nothing more than a typical guitar/bass amplifier. Big venues need a lot more hardware than this.
- demondemidi 2y ago“Big venues”. lol. I’m sure when Metallica played in russia in 1991 for the 4th largest concert in history it didn’t really happen because according to you they needed digital (glitchy) hardware. What are you even talking about, man.
- shiroiushi 2y agoThey don't have the old stuff any more. It's like trying to land men on the Moon now: the Apollo hardware is all gone, except the stuff in a museum that doesn't work any more.
- demondemidi 2y agoThat’s a good way to put it. But do mixing boards today really do ADC then DAC?!? Seems utterly pointless.
- shiroiushi 2y agoWhy? (Disclaimer: I'm not a professional in that industry, so I don't really know what I'm talking about.) Just as one example, the microphones and guitars these days are all connected to the mixing board through wireless radios, which we can assume are using digital protocols (just like all other wireless comms these days). Long gone are the days of ridiculously long cords that musicians trip over on stage. So at least that portion of the audio path has to be digital. And from what little I've been able to see of a modern mixing board from the audience, they look quite modern, with computerized displays, so of course they're going to be working with the audio digitally.
- zokier 2y agoOne of my pet peeves is that graphics programming does not generally enjoy this level of rigor like audio does, despite arguably being also real-time. This is doubly true for UI programming, I'm not aware of any UI toolkits designed with real-time in mind. I would love to see a UI system that has predictable low-latency real-time perf, so you could confidently achieve something like single frame latency on 144Hz display.
- kaba0 2y agoWell, if you allow a bit of a sloppy terminology, graphics are “softer” real-time, than audio - a frame drop is less noticeable than audio glitches.
- makapuf 2y agoThat's where realtime and fast differ : you can consider there's a "hard"-er real-time at 100ms by example, where you can drop some frames but you don't want to block the ui. Be slow but don't block the UI.
- Derbasti 2y agoOn the other hand, the throughput requirements on image processing are often far more stringent than in audio. A full-resolution stereo audio stream is only 200kb/s = 12 Mb/min after all, whereas a video system might have to chew throw many gigabytes in the same time.
- robinsonb5 2y agoWhen I last made a similar point someone pointed out to me that this has gained a higher priority recently thanks to VR. Janky framerate is going to be much more noticeable with a headset than with a traditional monitor, and could even result in lost lunches!
- dist-epoch 2y agoThe tiniest audio glitch is instantly noticed and extremely annoying. So extreme efforts are put into preventing them. If your software has audio glitches people will really stop using it. A graphics micro-stutter not so much. > I'm not aware of any UI toolkits designed with real-time in mind. What would be the point? The human eye can only notice so much FPS (gamers might disagree with their 244 FPS displays).
- forrestthewoods 2y ago> Applications where low latency is especially important are (1) interactive audio systems (such as musical instruments or DJ tools) where the UI needs to be responsive to the performer, and (2) real-time audio effects, where the system needs to process analog input (say from a guitar) and output the processed signal without noticeable delay. It's worth noting that these are practically the only case where extreme real-time audio programming measures are necessary. If you're making, for example, a video game the requirements aren't actually that steep. You can trivially trade latency for consistency. You don't need to do all your audio processing inside a 5ms window. You need to provide an audio buffer every 5 milliseconds. You can easily queue up N buffers to smooth out any variance. Highly optimized competitive video games average like ~100ms of audio latency [1]. Some slightly better. Some in the 150ms and even 200ms range. Input latency is hyper optimized, but people rarely pay attention to audio latency. My testing indicates that ~50ms is sufficient. Audio programming is fun. But you can inject latency to smooth out jitter in almost all use cases that don't involve a live musical instrument. [1] https://www.youtube.com/watch?v=JTuZvRF-OgE&t=490s https://www.youtube.com/watch?v=JTuZvRF-OgE&t=490s
- PaulDavisThe1st 2y agoShooting a gun (or whatever) in a game and "waiting" for the sound is extremely isomorphic to pressing a key on a MIDI keyboard and "waiting" for the sound. Yes, background sound in games can be handled with very large buffers, but most players expect music-performance-like latency for action-driven sound.
- forrestthewoods 2y agoGames don't have particularly large buffers. There's just a very long pipeline with lots of buffering. It's honestly pretty bad. But almost no one measures and it doesn't actually matter to players. Musicians have keenly trained ears. I would imagine their much more sensitive to audio latency than even a pro gamer, nevermind average Joe off the street. Where latency really matters is when you have a musical instrument that plays a sound and it's connected to a monitor. If those sounds are separated by more than 8ms or so the difference will be super noticeable to anyone, including Joe off the street. I'd be interested for someone to run a user study on MIDI keyboard latency. I'd bet $3.50 that anything under 40 milliseconds would be sufficient. Maybe 30 milliseconds. I'd be utterly shocked if it needed to be 8 milliseconds. And I'd be extremely shocked if every popular MIDI keyboard on the market actually hit that level of latency.
- deleted 2y ago[deleted]
- jcelerier 2y agoTimur Doumler's videos on the topic are also pretty good and bring some new methodologies to the table: https://youtu.be/zrWYJ6FdOFQ https://youtu.be/zrWYJ6FdOFQ https://youtu.be/vn7563IAQ_E https://youtu.be/vn7563IAQ_E https://youtu.be/7fKxIZOyBCE https://youtu.be/7fKxIZOyBCE
- ec109685 2y agoGood set of videos. Here is the article version of the first: https://timur.audio/using-locks-in-real-time-audio-processing-safely https://timur.audio/using-locks-in-real-time-audio-processin... The insight is that with two threads contending on one lock, there are efficient ways to build the lock that minimizes cpu on the non-realtime thread.