14 ms·
Four common mistakes in audio development (2016)
- pier25 7y agoPrevious discussion: https://news.ycombinator.com/item?id=11907887 https://news.ycombinator.com/item?id=11907887
- amelius 7y ago> Rendering live audio is quite demanding: the system has to deliver n seconds of audio data every n seconds to the audio hardware. If it doesn’t, the buffer runs dry, and the user hears a nasty glitch or crackle: that’s the hard transition from audio, to silence. I think you can do better when a buffer runs dry. Instead of outputting silence, you could output the same frequency spectrum as you were right before the event. That way you will not hear any cracks or pops. And of course you can fade out the effect when the buffer stays dry for more than a few seconds. Obviously, you'd have to do some filtering when the audio continues because that too can introduce cracks.
- kierenj 7y ago"Output the same frequency spectrum" - takes some considerable processing! We're usually working in time-domain, not frequency-domain. I think you're saying do a load of processing to fill the gap when you don't have time to do processing?
- amelius 7y agoAudio software often works in the frequency domain too. CPUs have optimized instructions for this (video also uses them). Also, the article speaks of multithreaded software, where deadlines can be missed because of complicated dependencies. The end stage where you correct for missing samples can work independently of them in its own thread.
- atoav 7y agoSo you always calculate the same frequency spectrum just in case of dropouts instead of getting the actual code right? Sorry, but if you ever find yourself in sich a situation stop, pause, make yourself a tea and consider how wise the thing you are doing really is.
- mbell 7y agoThere is really no such thing as 'instantaneous frequency response'. For any frequency to meaningfully exist, you need data for the corresponding period. e.g. If the audio contains content to 20hz, you need at least 1/40th to 1/20th of a second of data for that to materialize. Put another way - what you are proposing it looping the buffer which is what some devices do, portable CD players were kinda notorious for it and it doesn't sound much better than cracks or pops. Computers also have a tendency to fall into buffer looping when the system hangs (which is likely the result of the failure mode of realtek codecs).
- amelius 7y ago> There is really no such thing as 'instantaneous frequency response' Yes that's true, I'm proposing something that uses an approximation of it. Consider it from a different angle: the inner ear essentially performs a Fourier transform. At every moment the "instantaneous" spectrum determines which hair cells are triggered. Now what I propose is to keep triggering those same hair cells (and not any others) when the buffer runs dry. The exact way of accomplishing this is left as an exercise (though using short windows where you take a FFT could be a good approximation).
- human20190310 7y ago> The exact way of accomplishing this is left as an exercise Perhaps you should undertake this exercise and let us know how it sounds :) EDIT: In my experience with audio, when I have a bug that introduces even the slightest discontinuity (or even just a cusp) in the audio, well short of a pop to silence, I can still hear a "weirdness". Ears are pretty attuned to things that sound unnatural. I'm not confident that essentially "forging" the audio is going to sound natural.
- StavrosK 7y agoWhat if you train a deep neural network on the song so far, so it can generate plausible-sounding music whenever the buffer drops? You can even hang intentionally to generate original music! (/s, please don't)
- hashmal 7y agoAssuming it's a viable strategy with regards to processing resources (hint: it's not for anything more than toys) you will have audible artifacts, especially around transients. Filtering and additional processing will only alter the signal even more.
- hburd 7y agoI actually think it's actively harmful to hide problems that can be otherwise fixed. If the CPU is too busy to keep filling the audio buffer, the solution is to increase the buffer size to put less stress on the scheduler. I recently reduced my buffer size in Ableton Live, but I knew I had to increase it because I could hear pops. If these pops were being covered up, I wouldn't have realized my buffer size was too small and I'd be unknowingly introducing subtle artifacts into every recording.
- amelius 7y agoOk, but a large buffer size means more latency. Also, the settings you use for development do not have to be the same as those used in production.
- hunter2_ 7y agoBased on nothing more than user experience, conferencing software like Zoom tends to do something that sounds quite like what you're describing, complete with the fade to silence after about half a second. So it makes sense for certain live situations, but it wouldn't be desirable in studio recordings.
- vnorilo 7y agoFor the record, I think this is a terrible idea. However, the way I'd do it if needed: keep two or the most recent good buffers. When you need to synthesize, start running a phase vocoder based on the hop between those two. You get frozen sinusoids and some random noise for the bins that don't have one, and almost no cpu use on the happy path of no underruns. Still, don't do it :)
- amelius 7y ago> For the record, I think this is a terrible idea. It really depends on who you are asking. Some people just hate those loud cracks and pops, and would love to have something that filters them out naturally.
- rhizome 7y agoInstead of outputting silence, you could output the same frequency spectrum as you were right before the event. How does one detect whether it's a musical silence or a buffer underrun?
- nitrogen 7y agoYou'd probably do this at a layer where you have access to the buffer stats to know that the buffer is nearly empty.
- jcelerier 7y agoFor less Apple-centric guidelines : http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing http://www.rossbencina.com/code/real-time-audio-programming-...
- ris 7y agoAbsolutely, linux pro-audio developers have been handling these situations for over a decade now with JACK.
- shams93 7y agoYeah this is also why you see things like Reaper moving to linux and Bitwig debuting on Linux as well as mac and windows. With the future of mac and windows desktop computing up in the air I suggest that linux is the future for pro audio computing rather than apple.
- snvzz 7y agoJack is nearly useless without the linux-rt patchset and PREEMPT_RT. Mainline will have xruns once in a while even with large buffers (like 20ms). Do run cyclictest from rt-tests for a while if you don't believe this.
- bitwize 7y agoName me a working professional musician who doesn't use Apple gear (assuming they use computers at all to make music).
- deleted 7y ago[deleted]
- rocqua 7y agoSoftware defined radio deals with essentially the same requirements, and occurs much more on linux.
- osamagirl69 7y agoJoel Zimmerman, aka Deadmau5. You might see a macbook sitting on stage once in a while, but saying that he uses it to do his production would be no more accurate than saying he uses his watch to control the lights (he actually uses racks of nvidia GPUs)
- vemv 7y ago> Don’t use Objective-C/Swift on the audio thread. His argumentation is that Objective-C uses locks internally. But isn't it possible that those locks (as opposed to application-level locks) cause just negligible pauses, because there's no contention on them? I guess those internal locks exist to make ObjC's runtime dynamism feasible. One would expect the typical application to not leverage that dynamism (how many apps continously redefine methods under normal operation?).
- deleted 7y ago[deleted]
- jstsch 7y agoYou can use Objective C on the audio thread just fine in practice. Just don’t call out.
- adamnemecek 7y agoNo, you really can't. Show me a single audio app that uses objc in the render block.
- lfnoise 7y agoAgreed. You really must not. No sane developer would do it. Unfortunately, there are some insane developers.
- iainmerrick 7y agoIf there’s no contention, why do you think they’re there?
- deleted 7y ago[deleted]
- vemv 7y agoThey could be there as a guardrail for a worst-case scenario, without them being particularly needed for the average use case.
- adamnemecek 7y agoCurrent OS's are fundamentally unsuitable for audiowork. macOS < Windows < Linux < other OS. None of them really do real-time processing. It's really bad, I want a new OS where I can actually tell the scheduler what to do. Also, check out this music composition app I'm launching soon http://ngrid.io http://ngrid.io.
- tfolbrecht 7y agoWhy is the realtime Linux patchset + Jack audio server insuffecient for what you do?
- adamnemecek 7y agoA patchset is not an OS. Like I can make it work for myself but good luck making it work for users.
- iainmerrick 7y agoI thought you wanted it for yourself, though? If you wanted something that lots of other people could use, in principle you could make your own Linux distribution with a particular set of patches.
- adamnemecek 7y agoYou can't expect people to use your os.
- osamagirl69 7y agoThe linuxCNC project[1] is a distribution based on the real time kernel extensions for debian for real time cnc machine control. It used to be possible to get few-us guaranteed latency, but with modern multicore machines you are stuck with about 100us due to the way the caching works. Still plenty fast enough for audio! [1] http://linuxcnc.org/ http://linuxcnc.org/
- tfolbrecht 7y ago
- skybrian 7y agoI'm wondering if anyone here does embedded audio synthesis. Is a Teensy the way to go?
- moron4hire 7y agoIt's a little slow. I've used a Mega, but all around prefer the Adafruit Feather line.
- StavrosK 7y agoWhich processor, though? There are a few.
- moron4hire 7y agoI don't know if they've made something more updated, but I usually get the Feather M0 nrf52. The Bluetooth connectivity is super cool for controlling things, and as long as you don't enable encryption, it is relatively low latency.
- StavrosK 7y agoI see, thanks! I'm going to get a few to play with, I've seen other people recommend that microcontroller too.
- skybrian 7y agoThat's what I bought the first time, but unfortunately there don't seem to be good audio libraries for the Feather line, and the Teensy Audio library looks quite sophisticated. Or did I miss something?
- moron4hire 7y agoThey should be compatible. It's all Wiring at the end of the day. But IDK, I just write my own.
- 7y ago
- NieDzejkob 7y ago> It can be helpful to know that, on all modern processors, you can safety assign a value to a int, double, float, bool, BOOL or pointer variable on one thread, and read it on a different thread without worrying about tearing, where only a portion of the value has been assigned by the time you read it. What? NO! This is only relevant when you're writing assembly. In C, you have to use atomic (or possibly volatile, I'm not sure), or this just undefined behavior.
- coryrc 7y agoIf you are assigning something that fits in a single register, how would you get tearing? (Probably shouldn't have double in that list, in case it's in the special FPU registers)
- slx26 7y agoyeah, 64-bit types might cause problems (I found Rust's atomic module reference educative https://doc.rust-lang.org/std/sync/atomic/ https://doc.rust-lang.org/std/sync/atomic/ (the source code doesn't do anything too special either)), but the post said "modern processors", so I guess it's rather accurate
- cesarb 7y agoThe compiler is smarter than you think. Suppose one thread alternates between writing 32 and 33 to a 4-byte variable, while another thread alternates between writing 1024 and 1025 to the same 4-byte variable. You might think that the only possible values are 32, 33, 1024, 1025; but then you read the variable and find a 1056 or 1057 there. Why? Because the compiler noticed that the first thread, after its initial write of 32 or 33, only modifies the least significant byte of the value; so it generated a single-byte write to overwrite only the modified part of the value.
- joosters 7y agoI’d love to see a godbolt example of this optimisation, I’ve never heard of a compiler doing this sort of thing.
- cyberferret 7y agoAs an Audiobus user since pretty much day 1 (and also a Loopy app user), I'd just like to shout out to the author of this article with a "Thank You" for revolutionising music creation on Apple's small devices. I still remember the first time I managed to combine some of my favourite sound generator & synth apps and create a mix all on my iPad without having to export things to Garageband/Logic [0], and it was a real game changer on that platform. If anyone knows about audio development on iOS devices, this author is the 'go to' guy! [0] - https://soundcloud.com/cyberferret/industrial-ants-wav https://soundcloud.com/cyberferret/industrial-ants-wav
- jacobolus 7y ago> favourite sound generator & synth apps Any you’d specifically recommend?
- cyberferret 7y agoAbsolute favourite is Animoog - The interface is just so perfect for a touch screen device. Of course Loopy HD for looping. SampleTank is great for quickly creating beats and loops (and so is Figure for creating a random loop to build on). Bebot is just a heck of a lot of fun. I use Amplitube for a virtual guitar amp and it works really well with little lag. I've got a bunch of keyboard synth app, of which my faves are: Arturia iSEM, iMS-20 and Magellan. I know these are mainly old apps which have been around for a while, but I have hardly bought new apps these days because these ones have served me for years. For a sample of Animoog plus SampleTank - https://soundcloud.com/cyberferret/in-a-bad-moog https://soundcloud.com/cyberferret/in-a-bad-moog
- PascLeRasc 7y agoAudioKit Synth One is fantastic, and it's all open-source too.
- djmips 7y agoThanks for the link. To close the loop the audio kernel documentation in AudioKit Synth One references the OP. https://github.com/AudioKit/AudioKitSynthOne/blob/master/AudioKitSynthOne/DSP/README.md https://github.com/AudioKit/AudioKitSynthOne/blob/master/Aud... It's interesting in that it's written in Objective-C++ which in xcode are suffixed with .mm - This is for performance reasons I assume. I didn't know about Objective-C++ and here is a link about it. Is it kind of pushed into a dusty closet? Because I only found an archive.org link. https://web.archive.org/web/20101203170217/http://developer.apple.com/library/mac/#/web/20101204020949/http://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/ObjectiveC/Articles/ocCPlusPlus.html https://web.archive.org/web/20101203170217/http://developer....
- Nimitz14 7y agoLet's say you are recording and processing audio in realtime. You have one function available which runs on a separate thread and provides you with the samples (fixed number every call). As I understand this article (and similar ones), you are not supposed to malloc on that thread (as that thread hanging could result in dropped samples). But if you want to save the audio, how else are you going to save it without appending to some array (for which you need to malloc)?
- rabidrat 7y agopre-malloc the array.
- ComputerGuru 7y agoYou only need one result (the latest) at any given time, so you allocate room before your loop, set an atomic indicating that a calculation is in progress, then have exclusive rw access to that memory. It’s a really old trick, lots of unixy APIs returning strings or objects that you are not expected to free do a similar thing (function local static variables to storing results) at the cost of not being thread-safe.
- audio_dev 7y agoOne common solution is to have another thread continuously prepare a bunch of empty buffers in advance, the audio thread just uses one of them, and after that is filled another thread writes it to disk. No locking is required, and no malloccing/disk IO happens on the audio thread.
- echelon 7y agoCouldn't you just spin through buffers alloc'd at startup? Why do you need another thread for buffer management? You could do something like a ring buffer to change pointers. That said, I've just started working on my first audio pipeline app, and I'm having sampling rate issues making voices sound super deep.
- bsder 7y ago> But if you want to save the audio, how else are you going to save it without appending to some array (for which you need to malloc)? I don't know why everybody is making this so hard: You use a lock-free statically allocated circular buffer for all communication to or from a real-time audio thread. Period. Full stop. The audio thread wins under all contention scenarios. The audio thread ONLY ships data from the circular buffer to the hardware system or vice versa--IT DOES NO OTHER TASK. Everything else talks to the circular buffers. Nothing else works. The big questions are even if you do that--1) how do you keep the circular buffer from starving even with your other threads working to fill the circular buffer? and 2) if the circular buffer starves, what do you do?
- saagarjha 7y ago> Behind Objective-C’s message sending system (i.e. calling Obj-C methods) is some code that does a whole lot of stuff — including holding locks. Only if your method is not found in the IMP cache. Trying to acquire a lock for every objc_msgSend would be quite slow. > I decided not to mount an expedition into the source code because I didn’t know what to look for, or where to look for it, and I wasn’t even sure I’d even find it given that iOS and OS X are both pretty closed-off systems. So, we’ll have to take the word of those more knowledgeable than us, at least for now. macOS's libmalloc is open source (if not updated very often): https://opensource.apple.com/source/libmalloc/libmalloc-166.251.2/ https://opensource.apple.com/source/libmalloc/libmalloc-166....
- jjoonathan 7y agoThe post is about chasing the long tail of latency risk. A high -- even very high -- probability of not acquiring a lock is not good enough. Investigating libmalloc and discovering that somehow it had strictly bounded, suitable execution time wouldn't be good enough either, because it could plausibly change without notice.
- vnorilo 7y agoTo get an idea of the length of the tail: Suppose a deadline of 6ms to produce an audio buffer. If 0.01% of the callbacks choke on a lock, we get a dropout once a minute, which is quite a lot. This means we're interested in the 99.99th percentile.
- CamperBob2 7y agoLet’s say your GUI thread is holding a shared lock when the audio callback runs. In order for your audio callback to return the buffer on time it first needs to wait for your GUI thread to release the lock. Your GUI thread will be running with a much lower priority than the audio thread, so it could be interrupted by pretty much any other process on the system, and the callback will have to first wait for this other process, and then the GUI thread to finish and release the lock before the audio callback can finish computing the buffer — even though the audio thread may have the highest priority on the system. This is called priority inversion. I guess I fundamentally don't understand priority inversion. I've done a lot of realtime programming but have never had to confront that particular buzzword. If my realtime-priority I/O completion routine tries to take a lock that might be held by an idle-priority GUI thread, why is that a problem? If I'm waiting on a lock, then by definition I'm not running at realtime priority while I'm doing so. Thread priorities should apply only to threads that are actually doing stuff. It would be crazy for the kernel to fail to schedule any other threads while I'm waiting for one of them to release the lock I need. My realtime thread is effectively suspended during that wait, so it can't keep any others from running. Yes, I might have to wait for other threads and even other processes to be scheduled. Too bad. If I didn't want that to happen, I shouldn't have tried to take a lock in a realtime thread. What exactly should I have expected to happen?
- saagarjha 7y agoSomething else might start running instead of your realtime-priority thread or GUI thread.
- CamperBob2 7y agoYes, that's an occupational hazard when you're waiting on a lock. Someone else might do something. In fact, someone else had better do something, or you won't get your lock. IMHO, if your system is designed such that a GUI or other low-priority thread is contending for the same resources as a realtime I/O thread, in a situation where the lock might be held for so long that the I/O thread starves, then you have made a really elementary mistake... one so fundamental that it doesn't deserve an abstract-sounding term like "priority inversion." I still feel like there's something I'm missing here. I mean, I understand that the GUI thread in question might be heavily pre-empted by medium-priority threads, but eventually the GUI thread is going to be given enough time to finish whatever it was doing with my lock, and life will go on. Is this behavior somehow unexpected or counterintuitive? If kernels are designed such that lower-priority threads never get a time slice until higher-priority threads block on something or explicitly sleep or yield, well, that seems like a really foolish call on the part of the kernel designer. You literally might as well have a cooperative-multitasking OS at that point.
- amos19870630 7y agoExcellent article. Great explanation. Thanks Michael!
- deleted 7y ago[deleted]
- rstuart4133 7y agoWow, is all this really necessary now? Decades ago I recall programmers being immensely proud of their Z80 queue functions that didn't disable interrupts. They were needed for the same reasons the article spells out: dealing with hard real time events. But all it took to break it was another programmer leaving interrupts off we a wee bit too long in another piece of code, so we replaced these things with hardware queues that grew bigger and bigger over time - even UARTS had them. I am sort of amazed that audio recording hardware doesn't provide at least 100 milliseconds worth of buffering. It's all of 600 bytes of memory on a 48k bit/sec stream, so it would cost them nothing.
- Doxin 7y agoThe main problem with audio is that in many cases 100ms of latency is unacceptable. Having a large queue is a fine solution when latency isn't a big problem, but in a lot of cases for audio you want to get as close to no latency as possible.
- rstuart4133 7y agoThat's true in things like phone calls, but I did read the article looking for cases like that and didn't see any. It is all about post processing, where latency isn't an issue. All you need for that case is an accurate clock embedded in the stream. It really isn't rocket science, whereas here we have an entire article on keeping latency down in multithreaded code which should tell you it's both time consuming and requires an pretty skill full programmer. The rocket science bit starts after it's been written, and someone who doesn't have the big picture on where all the latencies are starts touching the code. The program doesn't break in an obvious way, or even deterministically. Instead the first symptom you are likely to notice is heisenbugs occurring in production and the frequency is gradually increasing as the code ages. Tracking that down and fixing it is rocket science. Compared to that the price of adding a buffer plus a clock in the next re-design is less than peanuts. We've been doing this stuff for decades now and gone through plenty of re-designs, which is why I'm amazed it hasn't happened.
- 7y ago