3 ms·

Pro audio plugins have lower latency and harder constraints than graphics processing. You cannot drop a frame, and roundtrip latency is ideally below 5ms. GPU A

by duped 4y ago

Pro audio plugins have lower latency and harder constraints than graphics processing. You cannot drop a frame, and roundtrip latency is ideally below 5ms. GPU Audio (the company) has a lot of hype but it's not really clear if they gain a ton. The existing analogs to GPUs are expensive pieces of special purpose DSPs like Waves Grid, Universal Audio's UAD, and the biggest which are Avid PT HDX accelerators. It turns out modern CPUs are plenty fast for heavy audio DSP. The point of friction is synchronization and the memory wall, most of the time. It turns out that audio DSP algorithms don't generally benefit from the same kind of parallelism you get on a GPU, since even the basic stuff you'd like to do (filtering, algorithmic reverb, etc) is a bunch of recursive/sequential algorithms that have a non-trivial cost to convert to a parallel form for SIMD evaluation, and the gains are not linear due to repeated computation (classic example, converting an exponential moving average which is used everywhere to N lanes of SIMD is a lot less than an N times speed up). That's not to say there isn't exciting work being done (GPU audio, for example). It's just really hard and not a magic bullet. It's not an Audio Processing Unit, after all. It also has video output, but not high fidelity audio i/o.

PaulDavisThe1st 4y ago
Wow. A low level, technical, detailed post about audio processing on HN that leaves me without feeling the need to correct any of it. Congratulations, and thanks!
colinsane 4y ago
GPUs have some use for physical modeling (e.g. convolutional reverb — which at least 5 years ago was not practical to do in realtime on a CPU). i toyed around with such a plugin a while back but i think it takes more expertise to program any reverb to sound obviously superior to the CPU-optimized algorithms we’re more practically familiar with. also additive synthesis greatly benefited from GPU acceleration at one point (particularly if you’re making full use of panning or detuning individual harmonics), but with SIMD/CPU improvements it’s much less relevant now. latency as a barrier to GPU use is sometimes overstated. you can reach sub-ms latency overhead on a GPU roundtrip. maybe relevant if you’re really targeting 5 ms through the whole pipeline, but not for amateur audio. of course, one issue is that these latencies stack if your host architecture uses something like VST/LV2 which only knows how to pass frames via the CPU. it’s one reason why i’d love for some declarative approach to plugins instead of shipping binaries which assume so much about their environment (even just shipping LLVM IR could let you batch GPU calls). but it seems that approach may forever be niche.
- duped 4y ago
  I haven't delved into it too deep. Some things are embarrassingly parallel, like additive synthesis. Some things can be made quite parallel like convolution. But a lot of stuff is not. The distinction I think that needs to be made between GPUs for graphics vs audio is that all GPU algorithms are built on the assumption that there exists a reasonable compute pipeline that every kind of processing can use or exploit. That is not true for audio processing (your compute kernels for convolution and IIR filtering are completely different, for example, even static IIRs and IIRs that undergo real time modulation have very different topologies). The thing I don't yet know about GPU Audio (the company) is how they deal with routing and process chains. Sub millisecond latency is fine, but if you have two dozen plugins in a chain (or god help you, a real DAG) then how do they handle the delay compensated I/O without adding extra latency for each sequential edge in the graph without the host being aware? It's not big things that hurt latency, it's more a death by a thousand cuts in practice.
- nyanpasu64 4y ago
  How practical is a "split reverb" with the low-latency parts computed on CPU and the high-latency parts computed on GPU (like how partitioned convolution is already performed)? Or is reverb/convolution not processor-intensive enough to reduce CPU workloads significantly?
  - duped 4y ago
    I don't think there's a meaningful difference between high and low latency segments of reverb algorithms except for some exotic hybrid models that use a mix of convolution for early reflections and FDNs for the longer tail, but the synchronization would absolutely suck to write and test
    - nyanpasu64 4y ago
      Naive convolution takes time proportional to the number of input samples multiplied by the length of the reverb impulse response (kernel), which becomes rather impractical for seconds-long reverb tails with >100k samples, convolved by ~50k samples per second. The usual solution is overlap-{add/discard} block convolution, where computational power decreases as the amount of input data processed at once increases (to a degree). But if you want short latency, you must process input data in small chunks, which increases CPU load. I think the trick of block convolution is to convolve the beginning of the reverb kernel in small chunks, and the later portions in large chunks (buffering small chunks of input into large chunks, convolving large chunks once available, and playing the result back in small chunks), and adding the two reverb tails (though I haven't fully studied or tried implementing it myself, and IIRC it's patented anyway).

by duped 4y ago

PaulDavisThe1st 4y ago

Wow. A low level, technical, detailed post about audio processing on HN that leaves me without feeling the need to correct any of it. Congratulations, and thanks!

colinsane 4y ago

GPUs have some use for physical modeling (e.g. convolutional reverb — which at least 5 years ago was not practical to do in realtime on a CPU). i toyed around with such a plugin a while back but i think it takes more expertise to program any reverb to sound obviously superior to the CPU-optimized algorithms we’re more practically familiar with. also additive synthesis greatly benefited from GPU acceleration at one point (particularly if you’re making full use of panning or detuning individual harmonics), but with SIMD/CPU improvements it’s much less relevant now. latency as a barrier to GPU use is sometimes overstated. you can reach sub-ms latency overhead on a GPU roundtrip. maybe relevant if you’re really targeting 5 ms through the whole pipeline, but not for amateur audio. of course, one issue is that these latencies stack if your host architecture uses something like VST/LV2 which only knows how to pass frames via the CPU. it’s one reason why i’d love for some declarative approach to plugins instead of shipping binaries which assume so much about their environment (even just shipping LLVM IR could let you batch GPU calls). but it seems that approach may forever be niche.

duped 4y ago
I haven't delved into it too deep. Some things are embarrassingly parallel, like additive synthesis. Some things can be made quite parallel like convolution. But a lot of stuff is not. The distinction I think that needs to be made between GPUs for graphics vs audio is that all GPU algorithms are built on the assumption that there exists a reasonable compute pipeline that every kind of processing can use or exploit. That is not true for audio processing (your compute kernels for convolution and IIR filtering are completely different, for example, even static IIRs and IIRs that undergo real time modulation have very different topologies). The thing I don't yet know about GPU Audio (the company) is how they deal with routing and process chains. Sub millisecond latency is fine, but if you have two dozen plugins in a chain (or god help you, a real DAG) then how do they handle the delay compensated I/O without adding extra latency for each sequential edge in the graph without the host being aware? It's not big things that hurt latency, it's more a death by a thousand cuts in practice.
nyanpasu64 4y ago
How practical is a "split reverb" with the low-latency parts computed on CPU and the high-latency parts computed on GPU (like how partitioned convolution is already performed)? Or is reverb/convolution not processor-intensive enough to reduce CPU workloads significantly?
- duped 4y ago
  I don't think there's a meaningful difference between high and low latency segments of reverb algorithms except for some exotic hybrid models that use a mix of convolution for early reflections and FDNs for the longer tail, but the synchronization would absolutely suck to write and test
  - nyanpasu64 4y ago
    Naive convolution takes time proportional to the number of input samples multiplied by the length of the reverb impulse response (kernel), which becomes rather impractical for seconds-long reverb tails with >100k samples, convolved by ~50k samples per second. The usual solution is overlap-{add/discard} block convolution, where computational power decreases as the amount of input data processed at once increases (to a degree). But if you want short latency, you must process input data in small chunks, which increases CPU load. I think the trick of block convolution is to convolve the beginning of the reverb kernel in small chunks, and the later portions in large chunks (buffering small chunks of input into large chunks, convolving large chunks once available, and playing the result back in small chunks), and adding the two reverb tails (though I haven't fully studied or tried implementing it myself, and IIRC it's patented anyway).