3 ms·

Comcast reduced “working latency” 90% by deploying AQM during Covid

20 points by dtaht 5y ago

winternett 5y ago
I got Comcast internet in September, the 200MBPS plan for 50$ per month and for the first 2 months speed tests averaged 150MBPs down and 6MBPS up... For the first 2 months, then speed degraded to 50MBPs down and 3MBPS up... somehow, and isn't going back up. Criminals. The main reason why I didn't have much choice was that Verizon was out of stock on routers and I needed a connection immediately. Verizon was more expensive anyway, and their plans were tied to specific hardware, which would make it difficult/more costly to upgrade easily later on. I also had an order in for T-Mobile's home internet at $50 per month where speed tests averaged 50MBPs down and 3MBPS up... I personally believe that all services are seeking advantage with any customer plan, so my solution is to keep both services and just switch to with network that is more reliable and as an end result be able to have Internet that is less dependent on a single point of failure with my Verizon phone as a potential back-up hotspot. Plain and simple, it's fleecing and theft that Government has permitted over a long span of time that will not change. All of the plans and terms of service are changed at the will of service providers. They learned their tactics from mobsters and drug dealers... Thoroughly convinced that possibly some decedents of crime organizations run a lot of these companies to be honest. May your costs be low and your speed high my friends... Ay... :/
nickysielicki 5y ago
This is damning evidence that packet shaping is completely necessary and that net neutrality is unrealistic. All packets are not created equally: zoom packets for client A or Call of Duty packets for client B should go ahead of a 30s chunk of Netflix video for client C that has 60 seconds of video in the tank. The net neutrality solution to this problem is to spend billions of dollars building out redundant and wide pipes so that everyone can go fast all the time. Comcast solved it with router configuration.
- traverseda 5y ago
  There's a way to do it in a way that doesn't completely violate net neutrality. Make some mechanism that programs can use to mark connections as bulk-rate or priority. Limit people's priority bandwidth to maybe 10% of their rated max throughput. That requires some new technology, but letting the user choose when a connection should be considered a low-latency priority is important to keep this kind of system from turning into a bunch of consumer-unfriendly back-room dealing. Creating an opaque system with unclear rules about what connections are classified as what priority level will also make it very difficult for new businesses to get their start, as eventually I think you'll probably only be considered high priority if you're a big business or you're mimicking the characteristics of some big businesses application.
  - willcipriano 5y ago
    Most people don't use the whole pipe, I doubt I use 10% of my total available traffic each month and I'm a power user. At 100 Mbit/s you'd have 100 GB per day of high priority traffic to use.
    - traverseda 5y ago
      I mean I think imagining it as 100GB of high priority traffic per day is the wrong way to think about it. I know that in some places in the US for some reason you pay for the bandwidth you use, instead of for the capacity of your line. Personally I think that's silly. With a 100Mbits/s connection you'd have 10Mbits/s of high priority traffic, per second. I feel like an ISP should be able to not completely oversubscribe to the point where you can't even use 10% of your connection at low-latency.
      - dtaht 5y ago
        by multiplexing better, you generally need to not differentiate between high and low priority. Being that I've been trying to convince people of this for years - all I can do is try to point at resources that demonstrate how we think about how packets work is frequently wrong, and/or try to provide a demonstrable example. Let's take voip vs netflix over 60ms - netflix, like most DASH video traffic from most video services, is very bursty. It tries to grab a chunk of video, and over the course of 1-2 seconds, fills the pipe, and voip sends one tiny packet every 20ms, which kind of looks like this: What you want: NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN -> you V V V What you get: VVVNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN -> you What AQM does is keep that induced queue short (roughly 16ms with DOCSIS-PIE). The netflix "pulse" goes away and you end up with NNNNNNNNNNNVNNNNNNNNNNNNNN -> you Netflix loads the next segment a tiny fraction slower, but your voip call doesn't get the jitter and latency side-effect anymore.
- dtaht 5y ago
  AQM technologies were described as necessary best current practice by the IETF back in 1992, and recommendations revised in 2013 https://datatracker.ietf.org/doc/rfc7567/ https://datatracker.ietf.org/doc/rfc7567/ because the algorithm for it chosen in 1992 (RED) didn't work well enough. Two new algorithms appeared in 2012 (Pie and codel/fq-codel) that did. I was very frustrated by the network neutrality "debate", partisans excluding the idea that it was actually a huge technical flaw with how the internet is structured that was at the root of the problem - when many flows attempted to co-exist: https://blog.cerowrt.org/post/net_neutrality_customers/ https://blog.cerowrt.org/post/net_neutrality_customers/ Now Solved. Thoroughly. By those two theoretical breakthrus. Since, zillions of knowlegable users and new products (like those from openwrt, eero, google wifi, and many others) managed to fix the underlying bufferbloat problem for themselves via "smart queue management" or "optimizing for conferencing and gaming", but it required manual configuration and tweaking and the right place for better bandwidth does indeed lie within the ISPs shapers and CPE. And on by default. However, it's not as simple of "call of duty or zoom" going "ahead". Better multiplexing (FQ), and shorter queues (AQM) lead to those applications' packets mixing into the heavier netflix flows without manual intervention. Comcast

20 points by dtaht 5y ago

I got Comcast internet in September, the 200MBPS plan for 50$ per month and for the first 2 months speed tests averaged 150MBPs down and 6MBPS up... For the first 2 months, then speed degraded to 50MBPs down and 3MBPS up... somehow, and isn't going back up. Criminals. The main reason why I didn't have much choice was that Verizon was out of stock on routers and I needed a connection immediately. Verizon was more expensive anyway, and their plans were tied to specific hardware, which would make it difficult/more costly to upgrade easily later on. I also had an order in for T-Mobile's home internet at $50 per month where speed tests averaged 50MBPs down and 3MBPS up... I personally believe that all services are seeking advantage with any customer plan, so my solution is to keep both services and just switch to with network that is more reliable and as an end result be able to have Internet that is less dependent on a single point of failure with my Verizon phone as a potential back-up hotspot. Plain and simple, it's fleecing and theft that Government has permitted over a long span of time that will not change. All of the plans and terms of service are changed at the will of service providers. They learned their tactics from mobsters and drug dealers... Thoroughly convinced that possibly some decedents of crime organizations run a lot of these companies to be honest. May your costs be low and your speed high my friends... Ay... :/

nickysielicki 5y ago

This is damning evidence that packet shaping is completely necessary and that net neutrality is unrealistic. All packets are not created equally: zoom packets for client A or Call of Duty packets for client B should go ahead of a 30s chunk of Netflix video for client C that has 60 seconds of video in the tank. The net neutrality solution to this problem is to spend billions of dollars building out redundant and wide pipes so that everyone can go fast all the time. Comcast solved it with router configuration.

traverseda 5y ago
There's a way to do it in a way that doesn't completely violate net neutrality. Make some mechanism that programs can use to mark connections as bulk-rate or priority. Limit people's priority bandwidth to maybe 10% of their rated max throughput. That requires some new technology, but letting the user choose when a connection should be considered a low-latency priority is important to keep this kind of system from turning into a bunch of consumer-unfriendly back-room dealing. Creating an opaque system with unclear rules about what connections are classified as what priority level will also make it very difficult for new businesses to get their start, as eventually I think you'll probably only be considered high priority if you're a big business or you're mimicking the characteristics of some big businesses application.
- willcipriano 5y ago
  Most people don't use the whole pipe, I doubt I use 10% of my total available traffic each month and I'm a power user. At 100 Mbit/s you'd have 100 GB per day of high priority traffic to use.
  - traverseda 5y ago
    I mean I think imagining it as 100GB of high priority traffic per day is the wrong way to think about it. I know that in some places in the US for some reason you pay for the bandwidth you use, instead of for the capacity of your line. Personally I think that's silly. With a 100Mbits/s connection you'd have 10Mbits/s of high priority traffic, per second. I feel like an ISP should be able to not completely oversubscribe to the point where you can't even use 10% of your connection at low-latency.
    - dtaht 5y ago
      by multiplexing better, you generally need to not differentiate between high and low priority. Being that I've been trying to convince people of this for years - all I can do is try to point at resources that demonstrate how we think about how packets work is frequently wrong, and/or try to provide a demonstrable example. Let's take voip vs netflix over 60ms - netflix, like most DASH video traffic from most video services, is very bursty. It tries to grab a chunk of video, and over the course of 1-2 seconds, fills the pipe, and voip sends one tiny packet every 20ms, which kind of looks like this: What you want: NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN -> you V V V What you get: VVVNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN -> you What AQM does is keep that induced queue short (roughly 16ms with DOCSIS-PIE). The netflix "pulse" goes away and you end up with NNNNNNNNNNNVNNNNNNNNNNNNNN -> you Netflix loads the next segment a tiny fraction slower, but your voip call doesn't get the jitter and latency side-effect anymore.
dtaht 5y ago
AQM technologies were described as necessary best current practice by the IETF back in 1992, and recommendations revised in 2013 https://datatracker.ietf.org/doc/rfc7567/ https://datatracker.ietf.org/doc/rfc7567/ because the algorithm for it chosen in 1992 (RED) didn't work well enough. Two new algorithms appeared in 2012 (Pie and codel/fq-codel) that did. I was very frustrated by the network neutrality "debate", partisans excluding the idea that it was actually a huge technical flaw with how the internet is structured that was at the root of the problem - when many flows attempted to co-exist: https://blog.cerowrt.org/post/net_neutrality_customers/ https://blog.cerowrt.org/post/net_neutrality_customers/ Now Solved. Thoroughly. By those two theoretical breakthrus. Since, zillions of knowlegable users and new products (like those from openwrt, eero, google wifi, and many others) managed to fix the underlying bufferbloat problem for themselves via "smart queue management" or "optimizing for conferencing and gaming", but it required manual configuration and tweaking and the right place for better bandwidth does indeed lie within the ISPs shapers and CPE. And on by default. However, it's not as simple of "call of duty or zoom" going "ahead". Better multiplexing (FQ), and shorter queues (AQM) lead to those applications' packets mixing into the heavier netflix flows without manual intervention. Comcast