4 ms·

> [video] stuff has just become more and more complex and more and more distant from what is actually happening in hardware There's been a huge push recently t

> [video] stuff has just become more and more complex and more and more distant from what is actually happening in hardware There's been a huge push recently to remove layers from graphics APIs and expose hardware more directly, in the form of Vulkan, DX12, and Metal. The problem that this has uncovered is that graphics hardware is not only very complex but quite varied, so exposing "what is actually happening in hardware" is incompatible with portability and simplicity. As a result you get APIs that are either not portable or hard to use. > more and more processing is done by video hardware, and less and less by the CPU. Unfortunately this will remain a reality because it comes with a 20-100x performance advantage.

PaulDavisThe1st 6y ago
The "lots of different hardware" side of it is where audio hardware was 20 years ago. Devices with multiple streams in hardware, and without. Devices with builtin processing effect X, and without. Devices that could use huge buffer sizes and ones that could only use small ones. Devices that could use just 2 buffers, and devices that could use many. Devices that supported all sample rates in hardware, devices that only supported one. Things are not totally cleared up, but the variation has been much reduced, and as a result the design of the driver API and then user-space APIs above that have become more straightforward and more consistent even across platforms. >20-100x performance advantage. Yes, for sure this is one aspect of the comparison that really breaks down. Sure, there are audio interfaces (e.g. those from UAD) that have a boatload of audio processing capabilities on board, and offload that DSP to the hardware saves oodles of CPU cycles. But these are unusual, far from the norm, and not widely used. Audio interfaces have mostly converged on a massively less capable hardware design than they had 20 years ago.
- modeless 6y ago
  Yeah, most audio processing is well within CPU capabilities even with a 100x performance disadvantage, so there's just no need to reach for hardware acceleration and all the complexity that comes with it. Rendering cutting edge graphics, especially on modern 4k 120Hz displays, is well outside CPU capabilities so that 100x boost is unfortunately essential. Interestingly, for really advanced audio processing like sound propagation in 3D environments or physics-based sound synthesis [1], GPUs will probably be necessary. [1] https://graphics.stanford.edu/projects/wavesolver/ https://graphics.stanford.edu/projects/wavesolver/
  - PaulDavisThe1st 6y ago
    physically modelled synthesis is no more advanced than anything going on in a modern game engine for video, and in some senses, significantly less. so to restate something i said just below, the fact the "most audio processing is well within CPU capabilities" is only really saying "audio processing hasn't been pushed in the way that video has because there's been no motivating factor for it". if enough people were actually interested in large scale physically modelled synthesis, nobody would see "audio processing [as] well within CPU capabilities". why aren't there many people who want/do this? Well ... classic chicken and egg. The final market is no doubt smaller than the game market, so I'd probably blame lack of the hardware for preventing the growth of the user community. ironically, sound propagation in 3D environments is more or less within current CPU capabilities.
    - modeless 6y ago
      I agree that physically modeled synthesis could easily push audio processing requirements far beyond CPU limits and require hardware acceleration similar to graphics. But I disagree that it's a chicken and egg problem. It's a cost/benefit tradeoff and the cost of realtime physically modelled synthesis is higher than its benefit for most applications. Mostly because the alternative of just mixing prerecorded samples works so well, psychoacoustically. (Though I think VR may be the point where this starts to break down; I'd love to try a physically modeled drum set in VR.) I also doubt that specialized audio hardware could provide a large benefit over a GPU for physically modeled synthesis. Ultimately what you need is FLOPS and GPUs these days are pretty much FLOPS maximizing machines with a few fixed function bits bolted on the side. Swapping the graphics fixed function bits for audio fixed function bits is not going to make a huge difference. Certainly nothing like the difference between CPU and GPU.
      - PaulDavisThe1st 6y ago
        I suppose that I broadly agree with most of this. Only caveat would be that you also a very fast roundtrip from where all those FLOPS happen to the audio interface buffer, and my understanding is even today in 2020, current GPUs do not provide that (data has to come back to the CPU, and for understandable reasons, this hasn't been optimized as a low latency path). Re: physically modelled drums ... watch this (from 2008) !!! https://vimeo.com/2433260 https://vimeo.com/2433260 Consider the sound made right before the 3:00 mark when Randy just wipes his hand over the "surface". You cannot do this with samples :) It says something about the distance between my vision of the world and reality that even though I thought this was the most exciting thing I saw in 2008, there's almost no trace of it having any effect on music technology down here in 2020.

PaulDavisThe1st 6y ago

The "lots of different hardware" side of it is where audio hardware was 20 years ago. Devices with multiple streams in hardware, and without. Devices with builtin processing effect X, and without. Devices that could use huge buffer sizes and ones that could only use small ones. Devices that could use just 2 buffers, and devices that could use many. Devices that supported all sample rates in hardware, devices that only supported one. Things are not totally cleared up, but the variation has been much reduced, and as a result the design of the driver API and then user-space APIs above that have become more straightforward and more consistent even across platforms. >20-100x performance advantage. Yes, for sure this is one aspect of the comparison that really breaks down. Sure, there are audio interfaces (e.g. those from UAD) that have a boatload of audio processing capabilities on board, and offload that DSP to the hardware saves oodles of CPU cycles. But these are unusual, far from the norm, and not widely used. Audio interfaces have mostly converged on a massively less capable hardware design than they had 20 years ago.

modeless 6y ago
Yeah, most audio processing is well within CPU capabilities even with a 100x performance disadvantage, so there's just no need to reach for hardware acceleration and all the complexity that comes with it. Rendering cutting edge graphics, especially on modern 4k 120Hz displays, is well outside CPU capabilities so that 100x boost is unfortunately essential. Interestingly, for really advanced audio processing like sound propagation in 3D environments or physics-based sound synthesis [1], GPUs will probably be necessary. [1] https://graphics.stanford.edu/projects/wavesolver/ https://graphics.stanford.edu/projects/wavesolver/
- PaulDavisThe1st 6y ago
  physically modelled synthesis is no more advanced than anything going on in a modern game engine for video, and in some senses, significantly less. so to restate something i said just below, the fact the "most audio processing is well within CPU capabilities" is only really saying "audio processing hasn't been pushed in the way that video has because there's been no motivating factor for it". if enough people were actually interested in large scale physically modelled synthesis, nobody would see "audio processing [as] well within CPU capabilities". why aren't there many people who want/do this? Well ... classic chicken and egg. The final market is no doubt smaller than the game market, so I'd probably blame lack of the hardware for preventing the growth of the user community. ironically, sound propagation in 3D environments is more or less within current CPU capabilities.
  - modeless 6y ago
    I agree that physically modeled synthesis could easily push audio processing requirements far beyond CPU limits and require hardware acceleration similar to graphics. But I disagree that it's a chicken and egg problem. It's a cost/benefit tradeoff and the cost of realtime physically modelled synthesis is higher than its benefit for most applications. Mostly because the alternative of just mixing prerecorded samples works so well, psychoacoustically. (Though I think VR may be the point where this starts to break down; I'd love to try a physically modeled drum set in VR.) I also doubt that specialized audio hardware could provide a large benefit over a GPU for physically modeled synthesis. Ultimately what you need is FLOPS and GPUs these days are pretty much FLOPS maximizing machines with a few fixed function bits bolted on the side. Swapping the graphics fixed function bits for audio fixed function bits is not going to make a huge difference. Certainly nothing like the difference between CPU and GPU.
    - PaulDavisThe1st 6y ago
      I suppose that I broadly agree with most of this. Only caveat would be that you also a very fast roundtrip from where all those FLOPS happen to the audio interface buffer, and my understanding is even today in 2020, current GPUs do not provide that (data has to come back to the CPU, and for understandable reasons, this hasn't been optimized as a low latency path). Re: physically modelled drums ... watch this (from 2008) !!! https://vimeo.com/2433260 https://vimeo.com/2433260 Consider the sound made right before the 3:00 mark when Randy just wipes his hand over the "surface". You cannot do this with samples :) It says something about the distance between my vision of the world and reality that even though I thought this was the most exciting thing I saw in 2008, there's almost no trace of it having any effect on music technology down here in 2020.