3 ms·
An Ode to Renice
This is a story about renice
About a year ago I was having issues getting real-time audio to work and be acceptably in sync with real-time screencast streaming. I was running applications headless on Debian and capturing the output using built-in protocol commands for screencasting, but the audio output of these apps was not covered by the protocol and just being sent to the system sink.
So I built a little pulseaudio-based NodeJS server that used parec to listen to a loopback device to capture the audio output from these applications.
Initially I just streamed audio over an audio/mpeg mp3 endpoint with Connection: keep-alive and an HTML audio element on the client end. This caused issues tho because the streaming was habitually getting out of sync with the video.
Then I decided to chop up the audio into little packets on the serve, assemble it again on the client, and play it with the Web Audio API. This was much better!
Breaking the content into tiny chunks allowed the bandwidth capacity of the channel to be more efficiently utilized. Latency and jitter were greatly reduced, but it was still not good enough. We would get "drift" - a gradual increase in latency over time.
Thinking about it, it seemed like backpressure was the cause. Basically we were pumping as many little packets through the channel as we could produce, when we produced them, with scant care as to whether the receiver could accommodate those. So even tho chunking had improved latency, latency still increased over time, due to occasional congestion in the network, or at the receiver.
Things really hit a turn, and went next level when we introduced the concept of an "ACK". Basically flow signalling, that acknowledged receipt of a packet (or bundle of packets), and gave permission for the sender to transmit another 1 (or n) subsequent packets.
This nearly solved the latency issues. Gone was all drift, at the cost of the occasional packet loss. But as the audio packets were so infinitesimally small from a perceptual standpoint, there was not much impact on perceived audio quality. But by increasing sync between audio and video, perceived quality of service actually increased far more than was impacted by occasional packet loss.
However there was still an approximately 1 or 2 second delay that I could not remove. Try as I might I just couldn't figure out how to decrease this delay. I played with pulseaudio monitors and devices and parameters to no avail. It was really one of those "give up" technical challenges that doesn't budge to any solution.
Things continued like that for a few months, and then, one day, quite unassuming, the idea occured to me that maybe I should consider renicing some of the processes in the chain. I reniced the NodeJS server to a near real time priority. No change. I reniced the pulseaudio process and parec, and BOOM suddenly, the consistent 1 or 2 second lag (that had sometimes blown out randomly to much longer) was gone. Evaporated. Never to return.
I was ecstatic. It was hard to contain my excitement and I didn't try. It was one of those epic bug fixes that's crucial but part of a long and arduous process of trial and error.
And the solution turns out to be simple. Elegant. And OS-based, not disparaging at all, incredible that you can get that lever.
So this is not a lesson, it's a humble ode to the power and majesty of renice.
-------
additional notes in a comment
- keepamovin 3y agoNotes renice: https://stackdiary.com/linux-docs/renice/ https://stackdiary.com/linux-docs/renice/ audio server code: https://github.com/dosyago/BrowserBoxPro/blob/boss/src/services/instance/parec-server/index.js https://github.com/dosyago/BrowserBoxPro/blob/boss/src/servi... audio client code: https://github.com/dosyago/BrowserBoxPro/blob/boss/src/public/voodoo/src/constructor.js#L547 https://github.com/dosyago/BrowserBoxPro/blob/boss/src/publi... ------- FAQ What real time prio did you renice to? We tried a few around -16 but settled on -15. Did you try WebRTC for the audio? Yes, we tried the data channel for streaming the WAV chunks, but in this case WS was more reliable and faster (I guess it was because we were producing small chunks at a consistent rate). I'm interested in exploring WebRTC audio channel streaming if anyone knows, come and contribute: https://github.com/dosyago/BrowserBoxPro https://github.com/dosyago/BrowserBoxPro or get in touch at opensource@dosyago.com Why are you using WAV not MP3 for audio? Because we're chunking it. I didn't know how you can slice MP3 into tiny pieces for custom ACK-based streaming like we are doing, and I'm not even sure if it's possible: when I tried the MP3 became corrupted, but WAV worked fine, (I guess because it's linear not a compressed self-referential format like MP3). In tests the reduction in bandwidth due to MP3, was somewhat lost to the increase in latency / compute on the server to encode it. If anyone knows a better way to stream audio (or MP3) in this case come contribute: https://github.com/dosyago/BrowserBoxPro https://github.com/dosyago/BrowserBoxPro or get in touch at opensource@dosyago.com