4 ms·
It is fascinating to me, as someone who has been doing real time audio work for 20 years, to see how two fundamental processes have been at work during that tim
by PaulDavisThe1st 6y ago
It is fascinating to me, as someone who has been doing real time audio work for 20 years, to see how two fundamental processes have been at work during that time:
* audio developers recognizing the parallels between the way audio and video hardware works, and taking as many lessons from the video world as possible. The result is that audio data flow (ignoring the monstrosity of USB audio) is now generally much better than it used to be, and in many cases is conceptually very clean. In addition, less and less processing is done by audio hardware, and more and more by the CPU.
* video developers not understanding much about audio, and failing to notice the parallels between the two data handling processes. Rather than take any lessons from the audio side of things, stuff has just become more and more complex and more and more distant from what is actually happening in hardware. In addition, more and more processing is done by video hardware, and less and less by the CPU.
In both cases, there is hardware which requires data periodically (on the order of a few msec). There are similar requirements to allow multiple applications to contribute what is visible/audible to the user. There are similar needs for processing pipelines between hardware and user space code.
(one important difference: if you don't provide a new frame of video data, most humans will not notice the result; if you don't provide a new frame of audio fata, every human will hear the result).
I feel as if both these worlds would really benefit from somehow having a full exchange about the high and low level aspects of the problems they both face, how they have been solved to date, how might be solved in the future, and how the two are both very much alike, and really quite different.
- modeless 6y ago> [video] stuff has just become more and more complex and more and more distant from what is actually happening in hardware There's been a huge push recently to remove layers from graphics APIs and expose hardware more directly, in the form of Vulkan, DX12, and Metal. The problem that this has uncovered is that graphics hardware is not only very complex but quite varied, so exposing "what is actually happening in hardware" is incompatible with portability and simplicity. As a result you get APIs that are either not portable or hard to use. > more and more processing is done by video hardware, and less and less by the CPU. Unfortunately this will remain a reality because it comes with a 20-100x performance advantage.
- PaulDavisThe1st 6y agoThe "lots of different hardware" side of it is where audio hardware was 20 years ago. Devices with multiple streams in hardware, and without. Devices with builtin processing effect X, and without. Devices that could use huge buffer sizes and ones that could only use small ones. Devices that could use just 2 buffers, and devices that could use many. Devices that supported all sample rates in hardware, devices that only supported one. Things are not totally cleared up, but the variation has been much reduced, and as a result the design of the driver API and then user-space APIs above that have become more straightforward and more consistent even across platforms. >20-100x performance advantage. Yes, for sure this is one aspect of the comparison that really breaks down. Sure, there are audio interfaces (e.g. those from UAD) that have a boatload of audio processing capabilities on board, and offload that DSP to the hardware saves oodles of CPU cycles. But these are unusual, far from the norm, and not widely used. Audio interfaces have mostly converged on a massively less capable hardware design than they had 20 years ago.
- modeless 6y agoYeah, most audio processing is well within CPU capabilities even with a 100x performance disadvantage, so there's just no need to reach for hardware acceleration and all the complexity that comes with it. Rendering cutting edge graphics, especially on modern 4k 120Hz displays, is well outside CPU capabilities so that 100x boost is unfortunately essential. Interestingly, for really advanced audio processing like sound propagation in 3D environments or physics-based sound synthesis [1], GPUs will probably be necessary. [1] https://graphics.stanford.edu/projects/wavesolver/ https://graphics.stanford.edu/projects/wavesolver/
- PaulDavisThe1st 6y agophysically modelled synthesis is no more advanced than anything going on in a modern game engine for video, and in some senses, significantly less. so to restate something i said just below, the fact the "most audio processing is well within CPU capabilities" is only really saying "audio processing hasn't been pushed in the way that video has because there's been no motivating factor for it". if enough people were actually interested in large scale physically modelled synthesis, nobody would see "audio processing [as] well within CPU capabilities". why aren't there many people who want/do this? Well ... classic chicken and egg. The final market is no doubt smaller than the game market, so I'd probably blame lack of the hardware for preventing the growth of the user community. ironically, sound propagation in 3D environments is more or less within current CPU capabilities.
- aspaceman 6y agoCould you be a bit more tangible about how video developers have have just made things more complex? I don't necessarily disagree, since I can't wrap my mine around modern video codecs easily (in concept sure, but they're harder to use in practice), but that might be just because I'm ignorant?
- PaulDavisThe1st 6y agoI think Ralph (Levien) did a fine job of that in the original article, albeit primarily focused on the issues surrounding compositing. I wasn't talking about codecs, but about how software interacts with the hardware. We're already dealing with the move away from OpenGL, which for a while had looked as if it might actually finally wrap most video in a consistent, portable model. Metal and Vulkan are moderately consistent with each other, but less so with the existing OpenGL-inspired codebase(s). But there's even more simplistic levels than this: in audio, on every (desktop) platform, it is trivial for audio I/O to be tightly coupled to the hardware "refresh" cycle. WDM-WASAPI/CoreAudio/ALSA all make this easy, if you need it (and for low latency audio, you do). Doing this for graphics/video is an insanely complex task. I know of no major GUI toolkit that even exports the notion of sync-to-vblank let alone does so in any actually portable way (Qt and GTK have both made a few steps in this direction, but if you dive in deep, you find that its a fairly weak connection compared to the way things work in audio).
- dfox 6y agoHardware-wise both audio and video had converged on the exactly same thing: there is this buffer that should be scanned/played out. The fact that contents of this buffer is generated by bunch of specialized hardware for the video case and can be perfectly well computed in software for audio is just question of bandwidth and CPU performance.
- PaulDavisThe1st 6y agoThat's only half true. You still can't render an orchestral-size collection of physically modelled instruments on the CPU. A processor that could would be some of analogy for the modern GPU, but for various reasons, there has been little investment in chip design that would move further in this direction. Games in particular created a large push for specialized GPU architectures, and there's really no equivalent motivation to do the same for audio, in part because there's no standard synthesis approach that things would converge on (the way they have in GPUs). So, "perfectly well computed in software for audio" is more of a statement about the world as-it-is rather than one about any fundamental difference between the two media.
- pravus 6y agoIs it possible to do audio processing using a GPU? I don't have enough working knowledge or mathematics to fully comprehend the 3d pipeline but as I understand, it is a huge linear algebra/matrix manipulator and audio processing would fall into this. My reason for asking is that at least in the guitar world I've seen issues related to CPU processing and multiple VST plugins all wired together. Some guitarists use fairly complicated rigs to get their tone and when you have multiple amps, speaker cabinet simulation, and effects running you need a pretty powerful machine to do everything real-time without under-running buffers. There are specific hardware solutions for this but I've always wondered if you could just create shaders or something and dump the audio stream to the GPU and get back the transformed audio stream with minimal CPU burden. Any insight here would be very helpful to me.
- PaulDavisThe1st 6y ago
- munificent 6y agoI like your overall point a lot. I'm an ex-game developer who has also dabbled in audio programming and music so I see a little both sides. I think the key difference, though, is that the consumer video needs are probably in the area of two orders of magniture more computationally complex for video than for audio. On consume machines, almost no interesting real-time audio happens. It's basically just playback with maybe a little mixing and EQ. Your average music listener is not running a real-time software synthesizer on their computer. Gamers are actually probably the consumers with the most complex audio pipelines because you're mixing a lot of sound sources in real-time with low latecy, reverb, and other spatial effects. The only people doing real heavyweight real-time audio are music producers and for them it's a viable marketing strategy for audio programmers to expect them to upgrade to beefier hardware. With video, almost every computer user is doing really complex real-time rendering and compositing. A graphics programmer can't easily ask their userbase to get better hardware when the userbase is millions and millions of people. Also, of course, generating 1470 samples of audio per frame (44100 sample rate, stereo, 60 FPS) is a hell of a lot easier than 2,073,600 pixels of video per frame. I agree that audio pipelines are a lot simpler, but I think's largely a luxury coming from having a much easier computational problem to solve and a target userbase more willing to put money into hardware to solve it.
- m-hilgendorf 6y ago> The only people doing real heavyweight real-time audio are music producers I wouldn't discount the amount of people using voice assist and rely on noise reduction/echo cancellation in their meetings, or the near future where we have SMPTE 2098 renderers running on HTPCs. I really think we're only a few years away from seeing a lot more realtime DSP in consumer applications. Conferencing, gaming, and content consumption all benefit immensely, and it would good for us all to start thinking about the latency penalty on audio-centric HCI like we do for video.
- PaulDavisThe1st 6y agoYou don't generate 1470 samples of audio per frame. You generate N channels worth of 1470 samples per frame, and mix (add) them together. Make N large enough, and make the computation processes associated with generating those samples complex enough, and the difference between audio and video is not so different. Jacob Collier routinely uses 300-600 tracks in his mostly-vocal overdubs, and so for sections where there's something going on in all tracks (rare), it's more in the range of 400k-900k samples to be dealt with. This sort of track count is also typical in movie post-production scenarios. If you were actually synthesizing those samples rather than just reading them from disk, the workload could exceed the video workload. And then there's the result of missing the audio buffer deadline (CLICK! on every speaker ever made) versus missing the video buffer deadline (some video nerds claiming they can spot a missing frame :)