3 ms·
Twitch doesn’t need the same aggressive latency as Google Meet, but WebRTC is hard-coded to compromise on quality. In general, it’s quite difficult to customiz
by keepamovin 3y ago
Twitch doesn’t need the same aggressive latency as Google Meet, but WebRTC is hard-coded to compromise on quality.
In general, it’s quite difficult to customize WebRTC outside of a few configurable modes. It’s a black box that you turn on, and if it works it works. And if it doesn’t work, then you have to deal with the pain that is forking libwebrtc… or just hope Google fixes it for you.
Solving this was indeed quite a challenge, but what we did was marry WebRTC data channels with compressed video frames that were unencoded. This meant we could drop frames all day long with zero artefacts and seamlessly adapt to the changing bandwidth conditions you find in the real world.
The reason glitch-avoidance was so crucial was the same reason we wanted fine-grained control over the quality: the text and interface heavy use case would not permit frequent low quality frames.
Our ack-based protocol is simple and effective.
In fact, many companies use WebRTC data channels to avoid the WebRTC media stack (ex. Zoom).
So, in a sense we use an approach similar to Zoom. The post author says this didn't work for them because their datagrams were not efficiently chunked. Somehow, we don't worry about this, likely because we don't need to retransmit frames avoiding head-of-line blocking, and our frames are compressed with JPEG, making them smaller by default.
Our congestion control involves:
- per N frames ack (or ACK is worth N frames, more efficient than per-frame-ack)
- measuring frame-ack RTT over a short window and decreasing quality in 20% increments when alerts are tripped. As soon as RTT returns to normal we jump back to max quality.
- if quality hits minimum, we increase Q, where we normally send every Qth-frame.
Additionally we also maintain two channels: WebSocket and WebRTC, and we actually switch frames to the fastest one. It's usually WebRTC but sometimes it's WebSocket. This switching further eases congestion.
Overall, it's a pretty smoothly working machine, developed from scratch through thousands of iterations and experiments.
This system allows us to essentially side step most congestion.
We trade frame rate for latency, as staying in "sync" with real time is the most important metric for application usability and responsiveness. This is because in resource constrained scenarios it's more important for people to feel that their actions are still having immediate effects, than it is to receive high frequency visual updates about those effects.