4 ms·

I like your overall point a lot. I'm an ex-game developer who has also dabbled in audio programming and music so I see a little both sides. I think the key dif

I like your overall point a lot. I'm an ex-game developer who has also dabbled in audio programming and music so I see a little both sides. I think the key difference, though, is that the consumer video needs are probably in the area of two orders of magniture more computationally complex for video than for audio. On consume machines, almost no interesting real-time audio happens. It's basically just playback with maybe a little mixing and EQ. Your average music listener is not running a real-time software synthesizer on their computer. Gamers are actually probably the consumers with the most complex audio pipelines because you're mixing a lot of sound sources in real-time with low latecy, reverb, and other spatial effects. The only people doing real heavyweight real-time audio are music producers and for them it's a viable marketing strategy for audio programmers to expect them to upgrade to beefier hardware. With video, almost every computer user is doing really complex real-time rendering and compositing. A graphics programmer can't easily ask their userbase to get better hardware when the userbase is millions and millions of people. Also, of course, generating 1470 samples of audio per frame (44100 sample rate, stereo, 60 FPS) is a hell of a lot easier than 2,073,600 pixels of video per frame. I agree that audio pipelines are a lot simpler, but I think's largely a luxury coming from having a much easier computational problem to solve and a target userbase more willing to put money into hardware to solve it.

m-hilgendorf 6y ago
> The only people doing real heavyweight real-time audio are music producers I wouldn't discount the amount of people using voice assist and rely on noise reduction/echo cancellation in their meetings, or the near future where we have SMPTE 2098 renderers running on HTPCs. I really think we're only a few years away from seeing a lot more realtime DSP in consumer applications. Conferencing, gaming, and content consumption all benefit immensely, and it would good for us all to start thinking about the latency penalty on audio-centric HCI like we do for video.
PaulDavisThe1st 6y ago
You don't generate 1470 samples of audio per frame. You generate N channels worth of 1470 samples per frame, and mix (add) them together. Make N large enough, and make the computation processes associated with generating those samples complex enough, and the difference between audio and video is not so different. Jacob Collier routinely uses 300-600 tracks in his mostly-vocal overdubs, and so for sections where there's something going on in all tracks (rare), it's more in the range of 400k-900k samples to be dealt with. This sort of track count is also typical in movie post-production scenarios. If you were actually synthesizing those samples rather than just reading them from disk, the workload could exceed the video workload. And then there's the result of missing the audio buffer deadline (CLICK! on every speaker ever made) versus missing the video buffer deadline (some video nerds claiming they can spot a missing frame :)
- morsch 6y ago
  You're restating the grandparents point: The only people doing real heavyweight real-time audio are music producers. Once Jacob Collier is done rendering his song, it's distributed as a YouTube video with single stereo audio track. And, talking about learning lessons from other fields, there's no particular reason that Jacob has to to render his audio at full fidelity in real time. Video editors usually do their interactive work with a relatively low fi representation, and do the high fidelity rendering in a batch process that can take hours or days. As I'm sure you're aware.
  - PaulDavisThe1st 6y ago
    I don't know much about video editors. Film sound editors in movie post-production do not work the way you describe.
    - munificent 6y ago
      > Film sound editors in movie post-production do not work the way you describe. They don't generally lower fidelity. That is needed for video simply because the data sizes for video footage are so huge they are to work with. But DAWs do let users "freeze", "consolidate" or otherwise pre-render effects so that everything does not need to be calculated in real-time on the fly.
      - PaulDavisThe1st 6y ago
        I'm the original author of a DAW, so I'm more than familiar with what they do :) Film sound editors do not do what you're describing. They work with typically 600-1000 tracks of audio. They do not lower fidelity, they do not pre-render. Ten years ago, one of the biggest post-production studios in Hollywood used TEN ProTools system to be able to function during this stage of the move production process.
        munificent 6y ago
        Sure, but that's an extreme use case. Typical DAW users are using a pro-sumer machine and bouncing tracks as necessary to fit within their CPU limits.
dmit 6y ago
> Gamers are actually probably the consumers with the most complex audio pipelines because you're mixing a lot of sound sources in real-time with low latecy, reverb, and other spatial effects. There's a GDC talk out there about the complexities of audio in Overwatch, a competitive game. Not only do you have invaluable audio cues coming from 12 players at the same time, but also all of their combined abilities and general sound effects, like countdown timers and such. One of the bigger problems talked about was that in a competitive game you can't just scale the loudness of an opponent's footsteps by the direct distance between them and you. You need to take into account level geometry, and so the game has to raycast in order to determine how loud any single sound effect should sound to every active player in the game. Edit (link): https://www.youtube.com/watch?v=zF_jcrTCMsA https://www.youtube.com/watch?v=zF_jcrTCMsA
- bentcorner 6y ago
  One more thing to add about Overwatch - with all the attention they give to audio, if your CPU is pegged and the game is trying to render 3d audio, nothing can save it and it drops audio. The end effect is that on low-powered hardware your audio can end up buzzing. In addition the high costs of dolby atmos processing on the CPU affects the rest of the game, such that turning off dolby atmos is a viable way to get higher FPS at the cost of lower-fidelity audio.
sriku 6y ago
One big difference though is jitter tolerance. To some extent, dropping visual frames will be seen as a reduction in average frame rate. For example, if scroll stutters, people may brush it off as "oh yeah the machine's loaded right now" and forgive and forget. However, the tolerance level of audio glitches is much lower (I think for most, but certainly for me). So the leeway for a computation to bleed over from one frame to the next is far less than for video.
fomine3 6y ago
Now 3D Audio is a thing on PS5.

m-hilgendorf 6y ago

> The only people doing real heavyweight real-time audio are music producers I wouldn't discount the amount of people using voice assist and rely on noise reduction/echo cancellation in their meetings, or the near future where we have SMPTE 2098 renderers running on HTPCs. I really think we're only a few years away from seeing a lot more realtime DSP in consumer applications. Conferencing, gaming, and content consumption all benefit immensely, and it would good for us all to start thinking about the latency penalty on audio-centric HCI like we do for video.

PaulDavisThe1st 6y ago

You don't generate 1470 samples of audio per frame. You generate N channels worth of 1470 samples per frame, and mix (add) them together. Make N large enough, and make the computation processes associated with generating those samples complex enough, and the difference between audio and video is not so different. Jacob Collier routinely uses 300-600 tracks in his mostly-vocal overdubs, and so for sections where there's something going on in all tracks (rare), it's more in the range of 400k-900k samples to be dealt with. This sort of track count is also typical in movie post-production scenarios. If you were actually synthesizing those samples rather than just reading them from disk, the workload could exceed the video workload. And then there's the result of missing the audio buffer deadline (CLICK! on every speaker ever made) versus missing the video buffer deadline (some video nerds claiming they can spot a missing frame :)

morsch 6y ago
You're restating the grandparents point: The only people doing real heavyweight real-time audio are music producers. Once Jacob Collier is done rendering his song, it's distributed as a YouTube video with single stereo audio track. And, talking about learning lessons from other fields, there's no particular reason that Jacob has to to render his audio at full fidelity in real time. Video editors usually do their interactive work with a relatively low fi representation, and do the high fidelity rendering in a batch process that can take hours or days. As I'm sure you're aware.
- PaulDavisThe1st 6y ago
  I don't know much about video editors. Film sound editors in movie post-production do not work the way you describe.
  - munificent 6y ago
    > Film sound editors in movie post-production do not work the way you describe. They don't generally lower fidelity. That is needed for video simply because the data sizes for video footage are so huge they are to work with. But DAWs do let users "freeze", "consolidate" or otherwise pre-render effects so that everything does not need to be calculated in real-time on the fly.
    - PaulDavisThe1st 6y ago
      I'm the original author of a DAW, so I'm more than familiar with what they do :) Film sound editors do not do what you're describing. They work with typically 600-1000 tracks of audio. They do not lower fidelity, they do not pre-render. Ten years ago, one of the biggest post-production studios in Hollywood used TEN ProTools system to be able to function during this stage of the move production process.
      - munificent 6y ago
        Sure, but that's an extreme use case. Typical DAW users are using a pro-sumer machine and bouncing tracks as necessary to fit within their CPU limits.

dmit 6y ago

> Gamers are actually probably the consumers with the most complex audio pipelines because you're mixing a lot of sound sources in real-time with low latecy, reverb, and other spatial effects. There's a GDC talk out there about the complexities of audio in Overwatch, a competitive game. Not only do you have invaluable audio cues coming from 12 players at the same time, but also all of their combined abilities and general sound effects, like countdown timers and such. One of the bigger problems talked about was that in a competitive game you can't just scale the loudness of an opponent's footsteps by the direct distance between them and you. You need to take into account level geometry, and so the game has to raycast in order to determine how loud any single sound effect should sound to every active player in the game. Edit (link): https://www.youtube.com/watch?v=zF_jcrTCMsA https://www.youtube.com/watch?v=zF_jcrTCMsA

bentcorner 6y ago
One more thing to add about Overwatch - with all the attention they give to audio, if your CPU is pegged and the game is trying to render 3d audio, nothing can save it and it drops audio. The end effect is that on low-powered hardware your audio can end up buzzing. In addition the high costs of dolby atmos processing on the CPU affects the rest of the game, such that turning off dolby atmos is a viable way to get higher FPS at the cost of lower-fidelity audio.

sriku 6y ago

One big difference though is jitter tolerance. To some extent, dropping visual frames will be seen as a reduction in average frame rate. For example, if scroll stutters, people may brush it off as "oh yeah the machine's loaded right now" and forgive and forget. However, the tolerance level of audio glitches is much lower (I think for most, but certainly for me). So the leeway for a computation to bleed over from one frame to the next is far less than for video.

fomine3 6y ago

Now 3D Audio is a thing on PS5.