5 ms·

Most folks don't realize this, but many issues are actually due to bufferbloat. Get a powerful router that runs CAKE and you'll be amazed at how little jitter w

by vlan0 5y ago

Most folks don't realize this, but many issues are actually due to bufferbloat. Get a powerful router that runs CAKE and you'll be amazed at how little jitter will be seen while maximizing available bandwidth. Also, if you live in a dense urban area, you'll also likely suffer from issues with available wireless spectrum and have to compete with everyone's router blasting as loud as possible. https://www.bufferbloat.net/projects/ https://www.bufferbloat.net/projects/

gruez 5y ago
But isn't the problem at every level, including the modem? Having a powerful router wouldn't help if the modem it's connected has bufferbloat as well.
- Syonyk 5y ago
  If you've profiled the system and rate limit bandwidth to what's below the "steady state flow" of your network (including modem), then you never have buffers. It's usually only one or two layers you have to worry about before you get into some higher bandwidth backbones and the problem largely goes away. If L3's main links are buffering, the internet has some larger problems. As a concrete example here, I run networking for our church, and we've had to make a facility that really wasn't designed for livestreaming into something that can toss out a tolerable stream. We've got... no idea what we pay for, actually, but it reliably measures about 70/15 on Sunday mornings with nothing restricted. However, due to some various quirks of network naming, there are a lot of other devices on the network, some of which only get woken up for Sundays, and they like installing updates. Plus cell phones that recognize "Oooh, wireless!" I need to split the network up, but I also hate rejoining "things" to networks, and we have a few more of those than I really care to deal with. Experimentally, while our max upstream is about 15Mbit, things start to get erratic beyond about 10 - latency starts getting inconsistent and we start having stream issues. I've capped the livestream bandwidth at 8Mbit (we hardware transcode on site from the Main profile h.264 coming out of the switcher to High profile, at a lower bitrate), and that gets first priority - I'm using Mikrotik queues, so any packet coming out of the server on Sunday goes first. I played around with what everyone else gets, and eventually settled on around 2Mbit - that can mix in with our livestream and still not impact anything outbound. But I also had to cap our download. While the connection can do 70Mbit, I reliably saw upload dropping (even from 10Mbit) if something had pegged out download in the morning. So that's capped to a conservative 20Mbit, which is enough for most things, and is low enough to not interfere with the higher priority upload. This is all on timers, and the limits kick in Sunday before the services, and drop off afterwards. You can also find a lot of gains if your router prioritizes acks outbound. DSL modems were often so badly asymmetrical (think 25Mbit down, 768kbit up) that your upstream acks would end up in queues and not get through for a while (buffer bloat, though I didn't know that term at the time). You could radically improve a DSL modem's connection by putting some queues in to manage upload. Prioritize acks, and then limit your total upload to about 750kbit (on that example 768k modem) so that you weren't buffering in the modem. And all of this is entirely in the scope of the router, even if it's working around issues further upstream.
  - gruez 5y ago
    So you capped your connection from a theoretical 75/15 to 20/8? I guess it makes sense if you value latency above all else, but for situations where throughput matters (eg. streaming or downloading game patches), it's unacceptable. Dynamically setting your speed cap would get rid of this problem, but would be huge timesink to get it just right.
    - Syonyk 5y ago
      I leave 2Mbit for other uploads, so it's 20/10, but, yes. During times when we care about latency and upload packet loss with a "realtime" set of requirements (there's no mechanism to retransmit lost packets for livestreaming with what we're using, so lost packets are just lost and glitch), the connection is heavily restricted. However, if you're trying to reduce buffer bloat and random latency, the general concept of restricting to what your connection can actually tolerate works quite well. I've done it plenty on various connections over the years.
- vlan0 5y ago
  I said powerful router because running CAKE/fq_codel means your router's CPU will be processing each packet. Which is not normally the case for a router processing unicast traffic without an AQM algorithm. For instance, much of Ubiquiti's lineup will choke on anything over 500mbps w/ fq_codel (aka Smart Queue) enabled.
Bayart 5y ago
>Get a powerful router that runs CAKE It doesn't even need to be powerful if you're on a normal copper line for home use. I'm saying that running a 20€ Xioami router and having machines torrenting, an Android box watching stuff, phones watching YT etc. without hiccups.
matheusmoreira 5y ago
> Get a powerful router Any guide on how to do this properly? My ISP's hardware is garbage but I'm not sure how to replace it. Feels like it was easier in the DSL days.
- supertrope 5y ago
  https://www.teamfortress.tv/44322/shared-home-internet-without-lag-and-ping-spikes https://www.teamfortress.tv/44322/shared-home-internet-witho...

by vlan0 5y ago

gruez 5y ago

But isn't the problem at every level, including the modem? Having a powerful router wouldn't help if the modem it's connected has bufferbloat as well.

Syonyk 5y ago
If you've profiled the system and rate limit bandwidth to what's below the "steady state flow" of your network (including modem), then you never have buffers. It's usually only one or two layers you have to worry about before you get into some higher bandwidth backbones and the problem largely goes away. If L3's main links are buffering, the internet has some larger problems. As a concrete example here, I run networking for our church, and we've had to make a facility that really wasn't designed for livestreaming into something that can toss out a tolerable stream. We've got... no idea what we pay for, actually, but it reliably measures about 70/15 on Sunday mornings with nothing restricted. However, due to some various quirks of network naming, there are a lot of other devices on the network, some of which only get woken up for Sundays, and they like installing updates. Plus cell phones that recognize "Oooh, wireless!" I need to split the network up, but I also hate rejoining "things" to networks, and we have a few more of those than I really care to deal with. Experimentally, while our max upstream is about 15Mbit, things start to get erratic beyond about 10 - latency starts getting inconsistent and we start having stream issues. I've capped the livestream bandwidth at 8Mbit (we hardware transcode on site from the Main profile h.264 coming out of the switcher to High profile, at a lower bitrate), and that gets first priority - I'm using Mikrotik queues, so any packet coming out of the server on Sunday goes first. I played around with what everyone else gets, and eventually settled on around 2Mbit - that can mix in with our livestream and still not impact anything outbound. But I also had to cap our download. While the connection can do 70Mbit, I reliably saw upload dropping (even from 10Mbit) if something had pegged out download in the morning. So that's capped to a conservative 20Mbit, which is enough for most things, and is low enough to not interfere with the higher priority upload. This is all on timers, and the limits kick in Sunday before the services, and drop off afterwards. You can also find a lot of gains if your router prioritizes acks outbound. DSL modems were often so badly asymmetrical (think 25Mbit down, 768kbit up) that your upstream acks would end up in queues and not get through for a while (buffer bloat, though I didn't know that term at the time). You could radically improve a DSL modem's connection by putting some queues in to manage upload. Prioritize acks, and then limit your total upload to about 750kbit (on that example 768k modem) so that you weren't buffering in the modem. And all of this is entirely in the scope of the router, even if it's working around issues further upstream.
- gruez 5y ago
  So you capped your connection from a theoretical 75/15 to 20/8? I guess it makes sense if you value latency above all else, but for situations where throughput matters (eg. streaming or downloading game patches), it's unacceptable. Dynamically setting your speed cap would get rid of this problem, but would be huge timesink to get it just right.
  - Syonyk 5y ago
    I leave 2Mbit for other uploads, so it's 20/10, but, yes. During times when we care about latency and upload packet loss with a "realtime" set of requirements (there's no mechanism to retransmit lost packets for livestreaming with what we're using, so lost packets are just lost and glitch), the connection is heavily restricted. However, if you're trying to reduce buffer bloat and random latency, the general concept of restricting to what your connection can actually tolerate works quite well. I've done it plenty on various connections over the years.
vlan0 5y ago
I said powerful router because running CAKE/fq_codel means your router's CPU will be processing each packet. Which is not normally the case for a router processing unicast traffic without an AQM algorithm. For instance, much of Ubiquiti's lineup will choke on anything over 500mbps w/ fq_codel (aka Smart Queue) enabled.

Bayart 5y ago

>Get a powerful router that runs CAKE It doesn't even need to be powerful if you're on a normal copper line for home use. I'm saying that running a 20€ Xioami router and having machines torrenting, an Android box watching stuff, phones watching YT etc. without hiccups.

matheusmoreira 5y ago

> Get a powerful router Any guide on how to do this properly? My ISP's hardware is garbage but I'm not sure how to replace it. Feels like it was easier in the DSL days.

supertrope 5y ago
https://www.teamfortress.tv/44322/shared-home-internet-without-lag-and-ping-spikes https://www.teamfortress.tv/44322/shared-home-internet-witho...