6 ms·

I'm surprised that no one has mentioned bufferbloat yet. In most home routers, people can see seconds of delay when there's significant other traffic on the lin

I'm surprised that no one has mentioned bufferbloat yet. In most home routers, people can see seconds of delay when there's significant other traffic on the link. (And remember, "other traffic" can be ordinary web browsing, where pages average 2 mbytes these days...) The latency (and jitter) caused by bufferbloat has been a solved problem for almost five years: fq_codel, and the newer cake qdisc's are are in the Linux kernel, and people's home routers now could install LEDE/OpenWrt (https://lede-project.org https://lede-project.org). Two links: What can I do about bufferbloat? https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_about_Bufferbloat/ https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a... Test your network for bufferbloat: http://dslreports.com/speedtest http://dslreports.com/speedtest

ra1n85 10y ago
Bufferbloat is a side affect of congestion or microbursts, and not the primary cause of poor voice calls. Either packets get dropped (shallow buffer) or queued (enough buffer), and in both cases the quality of the call will suffer. Mitigation: Don't congest. If you need to, mark VOIP traffic with a DSCP value that has queues that are serviced prior to all others (e.g., EF, DSCP46).
- AstralStorm 10y ago
  Applications do it already, but edge routers are adept at ignoring these flags or applying them only to their own service. So that you use it and not competition. Net neutrality would be the antidote.
  - ra1n85 10y ago
    You're correct, most applications do. QoS/CoS services are best when clearly spelled out in service agreements. Providers may clear markings, but likely because either they don't want to have a free for all when it comes to their queues, or because they didn't purchase the gear that offers the rich queuing needed.
  - saurik 10y ago
    If those flags provide better quality of service, why am I not applying them to all of my traffic? Any attempt to "know" if my use is legitimate would undermine network neutrality.
    - richbhanover 10y ago
      Sure, you could make yourself crazy trying to prioritize all the traffic with varying kinds of flags. Or you could just let an algorithm (SQM - fq_codel or cake) automatically determine which flows are sending more than their fair share of traffic into the bottleneck link, and offer backpressure to slow those applications down, so other applications work well. re: net neutrality. That applies to whether my ISP (or any provider upstream in the Internet) should prioritize packets/data from various sources toward (or from) me. The answer should be, No. My contract with my ISP is that I get X mbps of data. The contract doesn't say the ISP will provide higher priority or data rate for service Y over service Z... SQM has no effect on net neutrality: I'm simply configuring my own router to prioritize the data that I'm requesting (or sending). My ISP should handle them in a totally neutral way...
- colanderman 10y ago
  Yes. I'm a Republic Wireless (WiFi VoIP phone) customer. They mark their RTP packets with DSCP 48 (CS6 / ToS 192) and their SIP packets with DSCP 56 (CS7 / ToS 224). I limit ingress & egress bandwidth on my home router (a Mikrotik), and prioritize flows with those DSCPs above all other traffic. The result is that even when my pipe is completely full, VoIP packet latency is practically unaffected. Compare to ~3 s latencies I would incur by Comcast's awful traffic shaping.
- pier25 10y ago
  I have almost no bufferbloat when downloading (about 1ms) but about 300ms when uploading. What does this mean?
  - ra1n85 10y ago
    I didn't look at that tool, but just reading this suggests that you're being shaped/policed more aggressively on egress (in the direction of your provider) than you are on ingress. Shaping and policing (typically) both use the same mechanisms as interface buffers to temporarily store packets in order to prevent congestion/discards. The distinction between them is that shaping actively buffers packets, where policing will more aggressively drop them (and aim to trigger a TCP congestion event) - in most cases, policing will involve a burst bucket that is somewhat similar to a shallow queue. This would make sense if you have a higher bit rate from your provider for down vs up.
  - colanderman 10y ago
    What speed connection do you have, and what speed connection did that tool report? If your connection is fast enough that that tool is unable to max it out for whatever reason (say you're on a poor WiFi connection), you will experience no bufferbloat.
- richbhanover 10y ago
  Yes, Don't Congest. But you can spend your life marking various kinds of traffic (many apps already do mark it, as someone down-thread mentioned), and pray that your router (and remote equipment) handles the markings properly. Or just use SQM with Cake qdisc. It actually looks at the "sojourn time" for packets in various streams/flows of data and offers backpressure for sessions that are generating queues of data: all the other flows "go right through" without a lot of other settings.
minikites 10y ago
I believe Ubiquiti's routers also implement fq_codel: https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-EdgeRouter-software-release-v1-7-0/ba-p/1287631 https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-E... > Add new "smart queue" feature providing FQ-CoDel + HTB function and this can be configured in the Web UI to provide better QoS experience for the WAN connection.
pimeys 10y ago
Thanks for these! I have the Omnia Turris with Openwrt at home, which has everything needed to fix this. I'd also hope that problems playing music in our home network from the NAS when moving big files in the lan could be solved with these settings.
lobster_johnson 10y ago
Anyone know if Mikrotik's RouterOS suffers from this?
- colanderman 10y ago
  Bufferbloat is typically a disease of your ISP (which throttles traffic), not of your router. A Mikrotik in its default configuration, like most other routers, will do nothing to address – nor exacerbate – bufferbloat caused by ISP throttling. However, it is fairly simple to address ISP bufferbloat using RouterOS's "simple queues". In the most basic configuration, just add a simple queue targeting your WAN interface, with bandwidth limits in each direction slightly (~5-10%) lower than what your ISP throttles you to. If you have a slower connection (<10 Mbps), select "default-small" as the queue type. That's it, bufferbloat solved. (You can get fancier with multiple queues or different queue types, but the above alone gives you 90% of the benefit.)
  - lobster_johnson 10y ago
    Awesome, thanks!
  - richbhanover 10y ago
    Not so - Bufferbloat can occur everywhere there's a bottleneck link. Your home router's link to the ISP is likely to be one, and most don't have any mitigation, and suffer from high latency. (And various ISP's equipment has bufferbloat for data coming toward your home.) Some sites suggest that you knock your bandwidth down to 70% of the link speed to "leave some headroom" for other packets. That's fine if you want to give away 30% of your capacity. SQM (implemented through fq_codel or cake qdisc's) takes off a few percentage points from the rated link speed, and achieves great results with minimal setup and configuration. Check out https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_about_Bufferbloat/ https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a... for more info.
    - colanderman 10y ago
      That is… exactly what I said? Bufferbloat occurs where there's congestion. That's generally on the ISP's router, since they're the ones throttling your connection to 10 Mbps or whatever. Yes, if you have a gigabit fiber connection and your home router can only handle 50 Mbps, you can get bufferbloat on your home router. But most people aren't in that scenario. SQM is great, but not available on RouterOS, which is what the GP was asking about.
Bombthecat 10y ago
Not in Germany, kernels and routers need! To be locked by law. I accidentally updated without know. Now I'm locked...
- detaro 10y ago
  citation needed. If that were true, quite a few router models available would be illegal, so it probably isn't. (TP-link started locking down at least some models, in response to FCC rules, and some manufacturers don't allow you to downgrade, but there is no legal requirement to do so and often ways around it. Other manufacturers like Linksys openly advertise their routers for use with OpenWRT)
  - Bombthecat 10y ago
    Oh boy, direct down vote instead of looking on your own or asking... http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX%3A32014L0053&from=EN http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELE... http://www.tp-link.de/download/Archer-C7_V2.html#Firmware http://www.tp-link.de/download/Archer-C7_V2.html#Firmware 1.For Archer C7(EU)V2. 2.The EU firmware was specialized for CE certification and can't be downgraded to other version, please click here for choosing your region and selecting the most suitable firmware version to upgrade. 3.Old firmware's configuration file can be imported into this new firmware; 4.Your device's configuration will be lost after upgrading, which means you need to configure your device again All routers are doing the same. EU law.
    - detaro 10y ago
      Sorry, was a bit to quick and I misread your post as describing the current situation, where this not (yet) applies. And last I've heard about it they extended the validity of the old rules for undefined time (due to most of the standards not being ready), but I can't find a good source for it now, so I might have gotten that wrong. My mistake for not checking better.
    - richbhanover 10y ago
      > All routers are doing the same. EU Law. Citation still needed. a) I reviewed your eur-lex.europa.eu link, and didn't find anything that prohibits installing other firmware. Please cite the chapter/article that's relevant. b) re: TP-Link firmware. It is true that TP-Link has added a technical means that makes it impossible to install certain other TP-Link firmware (the "downgrade" they mention). However, there are other firmware distributions (LEDE/OpenWrt/DD-WRT/others) that have the capability to install over the TP-Link factory firmware.

ra1n85 10y ago

Bufferbloat is a side affect of congestion or microbursts, and not the primary cause of poor voice calls. Either packets get dropped (shallow buffer) or queued (enough buffer), and in both cases the quality of the call will suffer. Mitigation: Don't congest. If you need to, mark VOIP traffic with a DSCP value that has queues that are serviced prior to all others (e.g., EF, DSCP46).

AstralStorm 10y ago
Applications do it already, but edge routers are adept at ignoring these flags or applying them only to their own service. So that you use it and not competition. Net neutrality would be the antidote.
- ra1n85 10y ago
  You're correct, most applications do. QoS/CoS services are best when clearly spelled out in service agreements. Providers may clear markings, but likely because either they don't want to have a free for all when it comes to their queues, or because they didn't purchase the gear that offers the rich queuing needed.
- saurik 10y ago
  If those flags provide better quality of service, why am I not applying them to all of my traffic? Any attempt to "know" if my use is legitimate would undermine network neutrality.
  - richbhanover 10y ago
    Sure, you could make yourself crazy trying to prioritize all the traffic with varying kinds of flags. Or you could just let an algorithm (SQM - fq_codel or cake) automatically determine which flows are sending more than their fair share of traffic into the bottleneck link, and offer backpressure to slow those applications down, so other applications work well. re: net neutrality. That applies to whether my ISP (or any provider upstream in the Internet) should prioritize packets/data from various sources toward (or from) me. The answer should be, No. My contract with my ISP is that I get X mbps of data. The contract doesn't say the ISP will provide higher priority or data rate for service Y over service Z... SQM has no effect on net neutrality: I'm simply configuring my own router to prioritize the data that I'm requesting (or sending). My ISP should handle them in a totally neutral way...
colanderman 10y ago
Yes. I'm a Republic Wireless (WiFi VoIP phone) customer. They mark their RTP packets with DSCP 48 (CS6 / ToS 192) and their SIP packets with DSCP 56 (CS7 / ToS 224). I limit ingress & egress bandwidth on my home router (a Mikrotik), and prioritize flows with those DSCPs above all other traffic. The result is that even when my pipe is completely full, VoIP packet latency is practically unaffected. Compare to ~3 s latencies I would incur by Comcast's awful traffic shaping.
pier25 10y ago
I have almost no bufferbloat when downloading (about 1ms) but about 300ms when uploading. What does this mean?
- ra1n85 10y ago
  I didn't look at that tool, but just reading this suggests that you're being shaped/policed more aggressively on egress (in the direction of your provider) than you are on ingress. Shaping and policing (typically) both use the same mechanisms as interface buffers to temporarily store packets in order to prevent congestion/discards. The distinction between them is that shaping actively buffers packets, where policing will more aggressively drop them (and aim to trigger a TCP congestion event) - in most cases, policing will involve a burst bucket that is somewhat similar to a shallow queue. This would make sense if you have a higher bit rate from your provider for down vs up.
- colanderman 10y ago
  What speed connection do you have, and what speed connection did that tool report? If your connection is fast enough that that tool is unable to max it out for whatever reason (say you're on a poor WiFi connection), you will experience no bufferbloat.
richbhanover 10y ago
Yes, Don't Congest. But you can spend your life marking various kinds of traffic (many apps already do mark it, as someone down-thread mentioned), and pray that your router (and remote equipment) handles the markings properly. Or just use SQM with Cake qdisc. It actually looks at the "sojourn time" for packets in various streams/flows of data and offers backpressure for sessions that are generating queues of data: all the other flows "go right through" without a lot of other settings.

minikites 10y ago

I believe Ubiquiti's routers also implement fq_codel: https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-EdgeRouter-software-release-v1-7-0/ba-p/1287631 https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-E... > Add new "smart queue" feature providing FQ-CoDel + HTB function and this can be configured in the Web UI to provide better QoS experience for the WAN connection.

pimeys 10y ago

Thanks for these! I have the Omnia Turris with Openwrt at home, which has everything needed to fix this. I'd also hope that problems playing music in our home network from the NAS when moving big files in the lan could be solved with these settings.

lobster_johnson 10y ago

Anyone know if Mikrotik's RouterOS suffers from this?

colanderman 10y ago
Bufferbloat is typically a disease of your ISP (which throttles traffic), not of your router. A Mikrotik in its default configuration, like most other routers, will do nothing to address – nor exacerbate – bufferbloat caused by ISP throttling. However, it is fairly simple to address ISP bufferbloat using RouterOS's "simple queues". In the most basic configuration, just add a simple queue targeting your WAN interface, with bandwidth limits in each direction slightly (~5-10%) lower than what your ISP throttles you to. If you have a slower connection (<10 Mbps), select "default-small" as the queue type. That's it, bufferbloat solved. (You can get fancier with multiple queues or different queue types, but the above alone gives you 90% of the benefit.)
- lobster_johnson 10y ago
  Awesome, thanks!
- richbhanover 10y ago
  Not so - Bufferbloat can occur everywhere there's a bottleneck link. Your home router's link to the ISP is likely to be one, and most don't have any mitigation, and suffer from high latency. (And various ISP's equipment has bufferbloat for data coming toward your home.) Some sites suggest that you knock your bandwidth down to 70% of the link speed to "leave some headroom" for other packets. That's fine if you want to give away 30% of your capacity. SQM (implemented through fq_codel or cake qdisc's) takes off a few percentage points from the rated link speed, and achieves great results with minimal setup and configuration. Check out https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_about_Bufferbloat/ https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a... for more info.
  - colanderman 10y ago
    That is… exactly what I said? Bufferbloat occurs where there's congestion. That's generally on the ISP's router, since they're the ones throttling your connection to 10 Mbps or whatever. Yes, if you have a gigabit fiber connection and your home router can only handle 50 Mbps, you can get bufferbloat on your home router. But most people aren't in that scenario. SQM is great, but not available on RouterOS, which is what the GP was asking about.

Bombthecat 10y ago

Not in Germany, kernels and routers need! To be locked by law. I accidentally updated without know. Now I'm locked...

detaro 10y ago
citation needed. If that were true, quite a few router models available would be illegal, so it probably isn't. (TP-link started locking down at least some models, in response to FCC rules, and some manufacturers don't allow you to downgrade, but there is no legal requirement to do so and often ways around it. Other manufacturers like Linksys openly advertise their routers for use with OpenWRT)
- Bombthecat 10y ago
  Oh boy, direct down vote instead of looking on your own or asking... http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX%3A32014L0053&from=EN http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELE... http://www.tp-link.de/download/Archer-C7_V2.html#Firmware http://www.tp-link.de/download/Archer-C7_V2.html#Firmware 1.For Archer C7(EU)V2. 2.The EU firmware was specialized for CE certification and can't be downgraded to other version, please click here for choosing your region and selecting the most suitable firmware version to upgrade. 3.Old firmware's configuration file can be imported into this new firmware; 4.Your device's configuration will be lost after upgrading, which means you need to configure your device again All routers are doing the same. EU law.
  - detaro 10y ago
    Sorry, was a bit to quick and I misread your post as describing the current situation, where this not (yet) applies. And last I've heard about it they extended the validity of the old rules for undefined time (due to most of the standards not being ready), but I can't find a good source for it now, so I might have gotten that wrong. My mistake for not checking better.
  - richbhanover 10y ago
    > All routers are doing the same. EU Law. Citation still needed. a) I reviewed your eur-lex.europa.eu link, and didn't find anything that prohibits installing other firmware. Please cite the chapter/article that's relevant. b) re: TP-Link firmware. It is true that TP-Link has added a technical means that makes it impossible to install certain other TP-Link firmware (the "downgrade" they mention). However, there are other firmware distributions (LEDE/OpenWrt/DD-WRT/others) that have the capability to install over the TP-Link factory firmware.