4 ms·

First of all: great work! Two questions. (1) Are there any benefits over H. 264/RTP/UDP in a point-to-point streaming scenario (like two non-stationary nodes

First of all: great work! Two questions. (1) Are there any benefits over H. 264/RTP/UDP in a point-to-point streaming scenario (like two non-stationary nodes directly connected via WiFi)? (2) If it was implemented on a fast enough hardware, are there any obstacles to achieving sub 20ms latency in a scenario like the one described in (1)? Thanks!

keithwinstein 7y ago
Thanks! (1) Salsify is mostly about how you control the video encoder (e.g., a VP8/VP9/AV1/H.264/H.265 encoder) and transport protocol (e.g., RTP/UDP). H.264, RTP, and UDP themselves don't say anything about the control part, i.e., how to (1) estimate the network path's varying capacity, and then (2) how to adapt the desired frame size to match that capacity, and then (3) how to actually encode a frame of video to match the desired compressed frame size or bitrate. If you have an unpredictable/variable network path, Salsify is probably better than the control strategies in systems with less tightly controlled encoders and transports, e.g. WebRTC.org/Chrome, Hangouts, Skype, or FaceTime. If the network is mostly constant or at least predictable, Salsify's not going to be helpful. So, bottom line is "maybe." (2) Getting 20ms p95 glass-to-glass latency with compressed digital video is pretty difficult even under ideal circumstances, and basically impossible with a consumer webcam. At 60 Hz, just the interval between two frames is already 17 ms! So if you have one frame's worth of buffer between the camera/USB/encoder/sender and a frame's worth of buffer in the receiver/video card/display, you're already toast. You would really have to work hard to pipeline everything. Even the very best gaming monitors have latencies on the order of 10 milliseconds, and it's hard to buy a USB camera that even starts giving you any bits from the picture before the exposure is over. (And v4l2 doesn't return from VIDIOC_DQBUF until the frame is completely received, so you're probably looking at changing the kernel if you want to use UVC.) So <20ms I think is really hard. DJI just released an end-to-end custom-engineered system that claims 28ms latency (https://www.dji.com/fpv/info#specs https://www.dji.com/fpv/info#specs). If you look at Figure 8(a) from the Salsify paper, Chrome (using the WebRTC.org codebase) is getting per-frame latencies on the order of 600 milliseconds even when the network is totally constant and perfect, and Salsify's per-frame latencies are around 250 milliseconds but less consistent. Then you have to factor in the extra latency associated with waiting for the next frame -- if these systems are running at 10 fps (look at the extreme sparsity of the dots after the network hiccup, especially for WebRTC), there's 100 ms of extra glass-to-glass latency right there. Getting to <20ms @p95 is just another world from where these programs are today.

keithwinstein 7y ago

Thanks! (1) Salsify is mostly about how you control the video encoder (e.g., a VP8/VP9/AV1/H.264/H.265 encoder) and transport protocol (e.g., RTP/UDP). H.264, RTP, and UDP themselves don't say anything about the control part, i.e., how to (1) estimate the network path's varying capacity, and then (2) how to adapt the desired frame size to match that capacity, and then (3) how to actually encode a frame of video to match the desired compressed frame size or bitrate. If you have an unpredictable/variable network path, Salsify is probably better than the control strategies in systems with less tightly controlled encoders and transports, e.g. WebRTC.org/Chrome, Hangouts, Skype, or FaceTime. If the network is mostly constant or at least predictable, Salsify's not going to be helpful. So, bottom line is "maybe." (2) Getting 20ms p95 glass-to-glass latency with compressed digital video is pretty difficult even under ideal circumstances, and basically impossible with a consumer webcam. At 60 Hz, just the interval between two frames is already 17 ms! So if you have one frame's worth of buffer between the camera/USB/encoder/sender and a frame's worth of buffer in the receiver/video card/display, you're already toast. You would really have to work hard to pipeline everything. Even the very best gaming monitors have latencies on the order of 10 milliseconds, and it's hard to buy a USB camera that even starts giving you any bits from the picture before the exposure is over. (And v4l2 doesn't return from VIDIOC_DQBUF until the frame is completely received, so you're probably looking at changing the kernel if you want to use UVC.) So <20ms I think is really hard. DJI just released an end-to-end custom-engineered system that claims 28ms latency (https://www.dji.com/fpv/info#specs https://www.dji.com/fpv/info#specs). If you look at Figure 8(a) from the Salsify paper, Chrome (using the WebRTC.org codebase) is getting per-frame latencies on the order of 600 milliseconds even when the network is totally constant and perfect, and Salsify's per-frame latencies are around 250 milliseconds but less consistent. Then you have to factor in the extra latency associated with waiting for the next frame -- if these systems are running at 10 fps (look at the extreme sparsity of the dots after the network hiccup, especially for WebRTC), there's 100 ms of extra glass-to-glass latency right there. Getting to <20ms @p95 is just another world from where these programs are today.