4 ms·

This is a good collection of recommendations. At a new job, I often end up making changes to internal queues that (it turns out) conform to these recommendation

by kqr 5y ago

This is a good collection of recommendations. At a new job, I often end up making changes to internal queues that (it turns out) conform to these recommendations, and get better performance and reliability out of it. Though I didn't learn it by engineering networks like the author -- I took the ideas from lean product development and applied them also to software. > Corollary: limit queue length to a statistically large burst (eg. 99th percentile). This is somewhat underspecified -- at what time frame are we calling it a burst? How much money/memory/resources are we willing to spend to maintain the queue? But, critically, are we willing to make a queue so long that we sacrifice throughput during a sustained overload? Sometimes it makes sense to handle only a 20 % burst or even less, because the shorter queues lead to nicer behaviour when the burst draws out into sustained overload.

wtallis 5y ago
Round-trip times provide an upper bound on what should be considered a burst, since one RTT is enough time to provide feedback to throttle a burst. Of course, different traffic flows through a bottleneck can have drastically different RTTs, so this isn't too helpful. In practice, CoDel works well for almost all current network technologies with its target parameter at the default of 5ms of allowable queueing delay. Given the structure of today's networks, an individual packet is unlikely to pass through more than a handful of congested bottlenecks, and a small multiple of 5ms of added delay is a tolerable worst-case comparable to the speed of light delays of long-distance connections. It's definitely somewhat unsatisfying to not have formal derivations of optimal parameters. But reasonable defaults that have been tested in the real world are still a huge improvement over the old way of having network devices that don't even attempt to handle congestion intelligently.
Dowwie 5y ago
I'm very interested in your approach applying lean product development patterns to network engineering.
- kqr 5y ago
  It's really nothing special, just some general principles that are sensible to me: - Production levelling/burst smoothing early in the process, so other steps don't have to bother with variability. - WIP constraints/limited queue sizes both to reveal problems and improve latency. - Kanban/backpressure to stop the problems at the source, and help troubleshooting. - Using Little's law as a guideline in tradeoffs between batch size and latency and size of system. - Deliberately shed load when it cannot be served in time, rather than vainly holding on to it for dear life because "surely we cannot outright reject work?!" - Dropping at the head of queues to improve latency and serve the fresher requests sooner. There's probably a lot more, and TFA covers some of it too.
dtaht 5y ago
Fair queueing makes a huge difference for a large percentage of traffic, and generally makes an AQM work better by better muxing packets. I have a long list of things that can be done listed here: http://www.taht.net/~d/broadcom_aug9_2018.pdf http://www.taht.net/~d/broadcom_aug9_2018.pdf A new one that has cropped up recently in terms of shortening queues, is rigorous application of the TCP_NOTSENT_LOWAT option, everywhere, but especially in containers.

by kqr 5y ago

wtallis 5y ago

Round-trip times provide an upper bound on what should be considered a burst, since one RTT is enough time to provide feedback to throttle a burst. Of course, different traffic flows through a bottleneck can have drastically different RTTs, so this isn't too helpful. In practice, CoDel works well for almost all current network technologies with its target parameter at the default of 5ms of allowable queueing delay. Given the structure of today's networks, an individual packet is unlikely to pass through more than a handful of congested bottlenecks, and a small multiple of 5ms of added delay is a tolerable worst-case comparable to the speed of light delays of long-distance connections. It's definitely somewhat unsatisfying to not have formal derivations of optimal parameters. But reasonable defaults that have been tested in the real world are still a huge improvement over the old way of having network devices that don't even attempt to handle congestion intelligently.

Dowwie 5y ago

I'm very interested in your approach applying lean product development patterns to network engineering.

kqr 5y ago
It's really nothing special, just some general principles that are sensible to me: - Production levelling/burst smoothing early in the process, so other steps don't have to bother with variability. - WIP constraints/limited queue sizes both to reveal problems and improve latency. - Kanban/backpressure to stop the problems at the source, and help troubleshooting. - Using Little's law as a guideline in tradeoffs between batch size and latency and size of system. - Deliberately shed load when it cannot be served in time, rather than vainly holding on to it for dear life because "surely we cannot outright reject work?!" - Dropping at the head of queues to improve latency and serve the fresher requests sooner. There's probably a lot more, and TFA covers some of it too.

dtaht 5y ago

Fair queueing makes a huge difference for a large percentage of traffic, and generally makes an AQM work better by better muxing packets. I have a long list of things that can be done listed here: http://www.taht.net/~d/broadcom_aug9_2018.pdf http://www.taht.net/~d/broadcom_aug9_2018.pdf A new one that has cropped up recently in terms of shortening queues, is rigorous application of the TCP_NOTSENT_LOWAT option, everywhere, but especially in containers.