4 ms·

> The problem isn't that your videoconferencing goes from 1080p to 360p -- it's that it stutters and stops working for seconds at a time, so students completely

> The problem isn't that your videoconferencing goes from 1080p to 360p -- it's that it stutters and stops working for seconds at a time, so students completely miss content, and introduces a lag Bufferbloat. Ideally, at least for TCP applications, with perfect network congestion control, reduced bandwidth due to streaming should only degrade data rate, but never massively increase the latency. A slight increase is normal, but a huge increase is not. Unfortunately, in real life, it's often the case, and this aspect is often overlooked by vendors, developers, and sysadmins, making things even worse than it should be. Network hardware or software is optimized for throughput, and its performance under heavy traffic is often not tested. One common practice is using a buffer that is as large as possible, so that you can push the data rate to the maximum and avoid dropping packets at all cost. This is called Bufferbloat and it's a latency disaster. When the upstream bandwidth is saturated, the downstream buffer keeps accepting more bytes/packets, effectively breaks the proper feedback signal used by congestion control algorithms, so it never kicks in in a timely manner, and the FIFO nature of the buffer means the latency of a saturated link is always as high as the buffer size. As long as something is using the bandwidth, the network will always be slow like a snail. When people start having this problem, they simply blame the bandwidth-intensive application, and set a QoS up to prioritize things like DNS and VoIP, and to punish downloaders (and big networks usually have incrediblely complex and elaborate rules), This appeared to "fix" the problem, but it does not solve the underlying problem, all it does is moving the problem to the low priority queue. Another trouble is unwanted buffer can exist everywhere, in applications, operation systems, and underlying hardware, and since there's a conflict between throughput and latency, and some buffers are even technically necessary (Wireless networks are the worst offender), there's still a long way to do to fix everything. I am only speaking from my experience of managing small LANs, but I got most of the information from Dave Täht and Jim Gettys, who have been working on this issue in the last 10 years. Gettys writes extensively on bufferbloat in his blog that states similar issues exist at a much larger scale, including the ISP edge and backbone. Perhaps the vast majority of the case we are seeing here is caused by a lack of bandwidth and not related to bufferbloat, but I guess bufferbloat is still here to blame for at least 20% of the cases. Here is a talk Dave Täht recently made: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/ https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-... And here is Gettys' blogpost: https://gettys.wordpress.com/2018/02/11/the-blind-men-and-the-elephant/ https://gettys.wordpress.com/2018/02/11/the-blind-men-and-th...

bouncycastle 7y ago
Bufferbloat.net
kd5bjo 7y ago
> Ideally, at least for TCP applications, with perfect network congestion control, reduced bandwidth due to streaming should only degrade data rate, but never massively increase the latency Fundamentally, TCP is designed to get all the bits to their destination. That’s great for file transfer, but not ideal for streaming video— at the transport level, there’s no way to decouple data rate from latency because it’s not allowed to drop anything. UDP is the latency-prioritized transport protocol for the internet. It drops packets that it can’t handle because they are expected to contain out-of-date information anyway. It’s generally mote complicated to use because all of the flow control needs to happen at the application layer.
- segfaultbuserr 7y ago
  > at the transport level, there’s no way to decouple data rate from latency because it’s not allowed to drop anything. Yes, fundamentally, TCP is not designed for applications with controlled latency. I only used TCP as an example here because TCP is supposed to provide a reasonable congestion and flow control mechanism at the protocol level, its behavior under congestion is well-specified, and suitable as an example. Unlike UDP, which depends on applications, as you said, > It’s generally mote complicated to use because all of the flow control needs to happen at the application layer. I'm actually aware of some UDP programs that deliberately send multiple packets in a row without any regards of flow control, winning extra throughput at the expense of other programs. On the other hand, the TCP specification includes Slow Start, Sliding Window, Congestion Control algorithms, and recently, Explicit Congestion Notification. A main design goal is the detection of available bandwidth in an end-to-end manner, so that the sender does not send at an unnecessarily high data rate which the network or the receiver is incapable of processing. Packet drops is usually required in TCP as a feedback signal to enable detection of mismatched bandwidth. It's not always optimal (a well-known example is wireless networks - a dropped packet often does not indicate mismatched bandwidth), and it doesn't have to use dropped packets as the feedback signal (TCP Vegas & TCP BBR use latency, ECN uses a status bit), but in practice, the vast majority of systems (BIC, Cubic, Veno, CTCP) do require it, and it represents 80%+ of the installation. > UDP is the latency-prioritized transport protocol for the internet. It drops packets that it can’t handle because they are expected to contain out-of-date information anyway. It's supposed to work in this way. Unfortunately, two issues exist, the first challenge is applying the congestion control techniques invented for Bufferbloat to UDP, this needs to be done in a per-app basis. Another issue is, Bufferbloat is not simply a breakdown of flow control, but the existence of the large, uncontrolled FIFO buffer in networking programs, drivers and devices without queue management with latency in mind, which leads to an inherent latency in a saturated link. It's equally detrimental to TCP and UDP, as it takes an unusually long time for any packet to leave the FIFO queue. Breaking protocol-level flow control is only an additional effect that keeps adding fuel to the fire. An earliest example was the large buffer in TX queue in DOCSIS modems, discovered, documented and reported by Jim Gettys, and fixed in a later DOCSIS specification. Another was the uncontrolled TX queue with 2000 frames in many Ethernet drivers that plagued early Linux kernels. At that time, fixing the problem in a LAN was as easy as, "ifconfig eth0 txqueuelen 1", effectively disabling it. Later, queue management was introduced and it's no longer a problem. But many devices are still broken, the only way to stop it, is to ensure the uncontrolled buffer never have a chance to be filled, either by upgrading the upstream bandwidth to be greater than one could ever use, or by traffic policing your own TX bandwidth. Then you can only cross your finger and hope there isn't another uncontrolled buffer in another piece of network equipment at downstream (usually works at home, where there's only a router and a PC that runs the latest Linux/BSD systems). And even when it's the case, TX buffers on the opposite direction are not under your control, and ISP edge can be a place for them to exist.

bouncycastle 7y ago

Bufferbloat.net

kd5bjo 7y ago

> Ideally, at least for TCP applications, with perfect network congestion control, reduced bandwidth due to streaming should only degrade data rate, but never massively increase the latency Fundamentally, TCP is designed to get all the bits to their destination. That’s great for file transfer, but not ideal for streaming video— at the transport level, there’s no way to decouple data rate from latency because it’s not allowed to drop anything. UDP is the latency-prioritized transport protocol for the internet. It drops packets that it can’t handle because they are expected to contain out-of-date information anyway. It’s generally mote complicated to use because all of the flow control needs to happen at the application layer.

segfaultbuserr 7y ago
> at the transport level, there’s no way to decouple data rate from latency because it’s not allowed to drop anything. Yes, fundamentally, TCP is not designed for applications with controlled latency. I only used TCP as an example here because TCP is supposed to provide a reasonable congestion and flow control mechanism at the protocol level, its behavior under congestion is well-specified, and suitable as an example. Unlike UDP, which depends on applications, as you said, > It’s generally mote complicated to use because all of the flow control needs to happen at the application layer. I'm actually aware of some UDP programs that deliberately send multiple packets in a row without any regards of flow control, winning extra throughput at the expense of other programs. On the other hand, the TCP specification includes Slow Start, Sliding Window, Congestion Control algorithms, and recently, Explicit Congestion Notification. A main design goal is the detection of available bandwidth in an end-to-end manner, so that the sender does not send at an unnecessarily high data rate which the network or the receiver is incapable of processing. Packet drops is usually required in TCP as a feedback signal to enable detection of mismatched bandwidth. It's not always optimal (a well-known example is wireless networks - a dropped packet often does not indicate mismatched bandwidth), and it doesn't have to use dropped packets as the feedback signal (TCP Vegas & TCP BBR use latency, ECN uses a status bit), but in practice, the vast majority of systems (BIC, Cubic, Veno, CTCP) do require it, and it represents 80%+ of the installation. > UDP is the latency-prioritized transport protocol for the internet. It drops packets that it can’t handle because they are expected to contain out-of-date information anyway. It's supposed to work in this way. Unfortunately, two issues exist, the first challenge is applying the congestion control techniques invented for Bufferbloat to UDP, this needs to be done in a per-app basis. Another issue is, Bufferbloat is not simply a breakdown of flow control, but the existence of the large, uncontrolled FIFO buffer in networking programs, drivers and devices without queue management with latency in mind, which leads to an inherent latency in a saturated link. It's equally detrimental to TCP and UDP, as it takes an unusually long time for any packet to leave the FIFO queue. Breaking protocol-level flow control is only an additional effect that keeps adding fuel to the fire. An earliest example was the large buffer in TX queue in DOCSIS modems, discovered, documented and reported by Jim Gettys, and fixed in a later DOCSIS specification. Another was the uncontrolled TX queue with 2000 frames in many Ethernet drivers that plagued early Linux kernels. At that time, fixing the problem in a LAN was as easy as, "ifconfig eth0 txqueuelen 1", effectively disabling it. Later, queue management was introduced and it's no longer a problem. But many devices are still broken, the only way to stop it, is to ensure the uncontrolled buffer never have a chance to be filled, either by upgrading the upstream bandwidth to be greater than one could ever use, or by traffic policing your own TX bandwidth. Then you can only cross your finger and hope there isn't another uncontrolled buffer in another piece of network equipment at downstream (usually works at home, where there's only a router and a PC that runs the latest Linux/BSD systems). And even when it's the case, TX buffers on the opposite direction are not under your control, and ISP edge can be a place for them to exist.