13 ms·

What Can I Do About Bufferbloat?

207 points by velmu 8y ago

isostatic 8y ago
Do you want to have jitter, or do you want loss? In reality I don't see buffer bloat on the internet adding jitter of more than a few milliseconds. I do see loss though or 20, 50, even 100ms. I'd rather have 50ms of jitter than 50ms of loss, but that's just my application.
- deleted 8y ago
  [deleted]
- Arie 8y ago
  Why would you have to choose either? fq_codel and cake use a tiny bit of packet loss to get a sending host to back off, for example to keep a large download flow within the limits of your home link. Other flows aren't affected. Bufferbloat regularly adds hundreds of ms on home internet connections, you can get an indication of your bufferbloat on http://www.dslreports.com/speedtest http://www.dslreports.com/speedtest
  - isostatic 8y ago
    Not on my connection. I would rather my UDP packet arrive 30ms late than not arrive - especially on high latency links where I want to process the packet before a 300ms round trip nack/retransmit has a chance to work. I don't see any buffer bloat or excessive jitter on my home internet (at least on wired connection) on BT ftth.
    - tgb 8y ago
      Note that the bufferbloat choice for me is whether or not I want my packets to arrive 1.5 seconds late, not 30ms.
  - doctorsher 8y ago
    Anecdotally, I've seen larger amounts of packet loss and jitter when TCP accelerates faster into a loss event. The small amount of preventative loss reduces both of these values.
- dtaht99 8y ago
  there is no such thing as 50ms of loss. Loss, when your queue is short, is recovered from in an round trip. Bloat on the internet can add seconds of jitter. ( http://www.dslreports.com/speedtest/results/bufferbloat http://www.dslreports.com/speedtest/results/bufferbloat) But, I too am allergic to loss. fq_codel supports ECN, (explicit congestion notification), which is enabled, now, universally by IOS. As near as I can tell, the ECN usage of 6% (https://www.ietf.org/proceedings/98/slides/slides-98-maprg-tcp-ecn-experience-with-enabling-ecn-on-the-internet-padma-bhooma-00.pdf https://www.ietf.org/proceedings/98/slides/slides-98-maprg-t... ) in france is almost entirely from free.fr's deployment of fq_codel which they enabled by default in 2012 (!!!!!!). I had expected all the ISPs to have lept on this by now....
  - isostatic 8y ago
    Yes there is. I have multiple UDP/RTP streams at different bitrates, from the same IP to the same IP. I then get a loss on a 30mbit stream (3000 packets a second) of 150 packets. At the exact same time I get a loss on a 20mbit stream of 100 packets, and a 10mbit stream of 50 packets. This is an outage for 150ms, probably because of a reroute in an MPLS network somewhere. My packets have already been emmitted by the time any round trip resend would have come back.
- chacham15 8y ago
  Tcp needs packets to be dropped quickly to get the feedback that your link can't handle the speed. Without that, it will appear that the speed of the network is fluctuating (as the buffer fills it appears your network is faster than it is, when the buffer is full it will start dropping packets which causes tcp to back off which causes the buffer to drain, now if tcp starts sending data before the buffer is empty you'll get a different apparent speed to tcp) and it is that fluctuation which causes the issues.
theogravity 8y ago
If you use Opnsense, I've written a tutorial on how you can configure it to reduce bufferbloat. https://forum.opnsense.org/index.php?topic=7423.0 https://forum.opnsense.org/index.php?topic=7423.0
vanburen 8y ago
I recently bought an Ubiquiti EdgeRouter X [1]. Which has a smart queue function that utilises fq_codel. Results from the DSLReports speed test [2], using the same router, the only difference being turning smart queue on: Smart queue off: https://i.imgur.com/zeY4rTd.png https://i.imgur.com/zeY4rTd.png Smart queue on: https://i.imgur.com/jfHpiFb.png https://i.imgur.com/jfHpiFb.png Notice the improvement in bufferbloat score from D to A+. Subjectively, I have noticed the connection seems more responsive and there no longer seem to be latency spikes when utilising all the upload bandwidth. It’s certainly worth considering bufferbloat, if you suffer from latency spikes when using all your upload bandwidth (I used to suffer from this a lot more, when I had an ADSL connection, with only 1 megabit upload). [1] https://www.ubnt.com/edgemax/edgerouter-x/ https://www.ubnt.com/edgemax/edgerouter-x/ [2] http://www.dslreports.com/speedtest http://www.dslreports.com/speedtest
- Arie 8y ago
  I love the ER-X because it's cheap, generally available and a fine router with SQM. Just keep in mind that with smart queue enabled it will top out at about 150-170Mbit. If you're on a faster connection the Edgerouter ER4 with its faster quad core CPU should be able to handle up to about 300-350Mbit. When you need to shape even more bandwidth, routers based on the Marvel XP Armada chipset do very well. I flashed a Linksys WRT1900ACS with OpenWRT and was able to shape about 600-750Mbps before it ran out of horsepower.
  - matwood 8y ago
    > I love the ER-X because it's cheap I love my ER-X, but once I got my gigabit connection it can't quite keep up compared to plugging directly into the modem. It's close enough that I'm not looking to replace it, but the next time I need a router I may go the NUC build your own route.
    - 0xffff2 8y ago
      How does one go about building a router from a NUC? Don't you need two NIC's for a router? (One for the modem, one to connect a switch for the LAN.) Or have enough people given up on wired networking that they just build routers where the entire LAN is on WiFi?
Arie 8y ago
An awful lot of home broadband connections suffer from bufferbloat and even for the ones that don't, a single host can easily hog all the bandwidth. If you're used to getting lag in your VoIP or gaming when a house mate starts a stream/download/torrent, this can be fixed :) The cake traffic shaper in OpenWRT is amazing for fighting bufferbloat in your home network and it can also do almost perfect fairness in dividing the available bandwidth per LAN host with very little configuration. Just get it as part of the SQM tools in OpenWRT and enable it. For the per-host-fairness take a look at the "Make cake sing and dance" from this link: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details https://openwrt.org/docs/guide-user/network/traffic-shaping/... If you use an Edgerouter, you can get the cake traffic shaper but you'll have to do without the easy web interface OpenWRT has: https://community.ubnt.com/t5/EdgeRouter/Cake-and-FQ-PIE-compiled-for-the-EdgeRouter-devices/td-p/1679844 https://community.ubnt.com/t5/EdgeRouter/Cake-and-FQ-PIE-com...
- graystevens 8y ago
  Thanks for the Edgerouter link, saved me doing some Googling - Off to got and apply this now and see what difference this makes to the Bufferbloat tests. As Arie mentions, this is a little more involved on the EdgeOS stuff, but doesn't look too complex for those that are used to a CLI or two. Edit: Entirely off-topic, but an equally interesting find – There is a WireGuard client for the EdgeRouter line: https://community.ubnt.com/t5/EdgeRouter/Release-WireGuard-for-EdgeRouter/td-p/1904764 https://community.ubnt.com/t5/EdgeRouter/Release-WireGuard-f...
  - voltagex_ 8y ago
    Ah crap, I wonder if I've done the wrong thing going with a switch + Security Gateway instead of an EdgeRouter.
    - andrewaylett 8y ago
      You get integration with the Unifi UI and a much easier configuration experience, you lose only because what you've got is (probably) overkill for residential use. If you've got more than one AP then you start to win again because you can power all your APs from the switch. If you were happy with your setup before hearing about the EdgeRouter, I hope you'll still be happy with it now :).
dtaht99 8y ago
Recently I ranted at the ISPs and at the network neutrality people. fq_codel (RFC8290) is now nearly ubiquitous in as the default queuing mechanism in most linux distributions, and it is long past time more ISPs supplied it in the gear they give customers. http://blog.cerowrt.org/post/net_neutrality_customers/ http://blog.cerowrt.org/post/net_neutrality_customers/ sch_cake (derived from fq_codel with an integral shaper) is about to enter the Linux mainline after years of baking in openwrt. https://lwn.net/SubscriberLink/758353/05c20f25115c852d/ https://lwn.net/SubscriberLink/758353/05c20f25115c852d/ We can end bufferbloat, now, on everything.
- pja 8y ago
  I enabled sch_cake on my router (an R7800 with Openwrt 18.06 on it) recently. Worked like a charm. Would happily recommend. I imagine the main thing that's holding back ISPs from using it on their consumer routers is the (minimal) configuration the user has to do to set to upload bandwidth limit. If ISP routers were also using decent queuing algorithms and not buffering like crazy then that wouldn't be necessary of course, but it doesn't seem like that's going to change any time soon.
  - dtaht99 8y ago
    Well, many isp-born devices are provided with that info by the ISP at connect time, so the hope has always been that during configuration they'd just pass (for example) "bandwidth 10mbit docsis" to the network setup routine. DSL modems also typically get this info at startup.
- philjohn 8y ago
  The problem is that ISP provided kit is built to a price, so will tend to have a more aenemic processor. Shaping high bandwidth needs CPU cycles, or better yet, hardware offloading. Funnily enough, I've just got a firmware applied to my cable connection with Virgin Media in the UK that mitigates the Puma 6 issue on their provided DOCSIS 3 modem - it's eliminated buffer bloat as well, which is a nice side effect. People are listening, Docsis 3.1 includes active queue management for instance, which goes a long way to preventing the issue.
snikeris 8y ago
News to me. dslreports reported a C for bufferbloat. My router (eero) has a "labs" section that had SQM disabled. After enabling, dslreports is reporting an A.
stuaxo 8y ago
Is there a list of routers and their support for the different queuing methods (and how they are setup by default) ?
- dtaht99 8y ago
  DOCSIS 3.1 devices mandate pie, which helps a lot, but it's not as good as fq_codel, nor do they do shaping from the isp, which is kind of needed for all the cable links in the USA I've tried. Get one if you can, though, they are better across the board in many other ways. pfsense has fq_codel. Anything (1000s of routers) from lede/openwrt has the most advanced bufferbloat-fighting stuff in it, followed by dd-wrt, tomato, etc. If you need high bandwidths the multi-core arms are the best. fq_codel is pre-configured on all links (ethernet/usb/fiber/wifi/whatever) but if you need shaping to the ISP provided rate you need to configure it. All the research that went into fixing bufferbloat queuing problems everywhere landed in openwrt and lede first. Most of the research that improved tcp everywhere came out of google. Most gaming routers now sold commercially have some variant of fq_codel in them in their trade name ISP "qos" system. Also fq_codel derived anti-bufferbloat work has landed in many commercial wifi routers on the wifi side (eero, google wifi, some ubnt products, meraki, many others). The paper behind all that was: https://arxiv.org/pdf/1703.00064.pdf https://arxiv.org/pdf/1703.00064.pdf - happily that work was "good enough" to enable by default, and boy, does it make a difference if wifi is your bottleneck. The current premier dsl router with cake is evenroute. I think there are several new models from several manufacturers that are going to get it right, soon. Not tracking FIOS (gpon fiber) closely at the moment. Yes, fiber networks have bufferbloat, but it's harder to hit, and generally smaller than on dsl and cable technologies. I configured cake on sonic fiber recently and got 60ms back. (going from 60ms latency under load to 3ms ) Regrettably shaper setup is finicky and requires a few minutes of testing with a site like dslreports or a tool like flent.org to get right. If more ISPs published their shapers' bitrate and burst rate settings, life would be easier here... but the hope has always been they'd just ship a router with this stuff on and remotely configured to be "right". In some ways, what are you doing about bufferbloat is
  - dtaht99 8y ago
    Let's see. The turris omnia is a very good router (but only available in europe). For oomph (gbit shaping) people often leverage lede on a pcengines apu2 or run a full distro of pfsense or linux on it.
josteink 8y ago
As someone who has meddled with QOS in the past, this page didn't really feel very concrete to me. Especially regarding SQM... Is SQM just a new name for QOS or am I missing something?
- rusk 8y ago
  Isn't QoS a more general qualitative, perceived phenomenon? SQM sounds like it relates to something you'd do to improve QoS but then only at one link in the chain, at one layer.
  - dtaht99 8y ago
    I wish! QoS could have been a good term to keep using if the existing deployments of it on the Internet it wasn't hopelessly mapped generally to mere packet prioritization (diffserv) which doesn't actually work on today's internet. QoE is a better, less overloaded, in the "qualitative, perceived" sort of description. So (after endless debates) - we defined SQM as as a superset of classic as-defined-for-internet QoS and hoped for the best. (https://www.bufferbloat.net/projects/cerowrt/wiki/Smart_Queue_Management/ https://www.bufferbloat.net/projects/cerowrt/wiki/Smart_Queu... ) There's been plethora of other trade names for what we do with htb+fq_codel - streamboost, adaptive qos, etc. I like that eero, edgerouter, and openwrt and derivatives also call what we do sqm. It simplifies the discussion, and the core scripts for linux generically are available as the sqm-scripts on github. I also hope we see more RFPs specifying RFC8290.
    - rusk 8y ago
      > deployments of it on the Internet it wasn't hopelessly mapped generally to mere packet prioritization (diffserv) Sorry yeah, I forgot about how the term has been mangled over the years. I was coming from a 3GPP perspective where the term is specifically defined as experience [0] In particular: * only the QoS perceived by end-user matter * QoS definitions have to be future proof; * QoS has to be provided end-to-end. * QoS attributes (or mapping of them) should not be restricted to one or few external QoS control mechanisms Not the first time the ivory tower of Telecomms has been out of touch with the outside world ... [0] 4.1 (p. 7) https://www.etsi.org/deliver/etsi_ts/123100_123199/123107/05.04.00_60/ts_123107v050400p.pdf https://www.etsi.org/deliver/etsi_ts/123100_123199/123107/05...
magoon 8y ago
I cured bhfferbloat with a cheap DD-WRT capable router (NOT Puma 6 chipset). I found it important to limit the WAN bandwidth on my router to about 15% below the max speed I’m paying for. What a difference! Netgear R6300V2 Using fq_codel because the others use more CPU than I’m confident these devices can handle. I find that most people suffer from bufferbloat unknowingly, having bought fancy routers 5-10 years ago.
scruffyherder 8y ago
Residential internet in Hong Kong can go as high as 10gig. I don't see anything about links anywhere around 10gig or higher, if I wanted this on my 100gig backbone or 40gig metro links.
- dtaht99 8y ago
  Bufferbloat happens on high speed links like those but amount of bloat you see is in the 30ms - 60ms range (vs seconds(!!) on home links). Bloat happens mostly (aside from microbursts) on overloaded links - and high speed backbones are typically overprovisioned so the problem only shows up when there's an outage or fiber cut. Example of what happens today on a fiber cut: http://blog.cerowrt.org/post/bufferbloat_on_the_backbone/ http://blog.cerowrt.org/post/bufferbloat_on_the_backbone/ IF things like fq_codel were deployed on those we'd not see latencies climb that much at all, we'd see bandwidths decline to the actual capacity available - and only the biggest flows would be hit to do so. FQ_codel is lightweight enough to fit directly in high speed hardware and it does indeed run on 40gigE plus devices on linux, ddpk, BSD, in software. But it takes a long time for new chipsets to adopt new algorithms even if they incorporate support for deeply desirable features like ECN. That said... 10Gbit to the home.... ooohhhhh. it's really hard to bloat that!
meta_AU 8y ago
Anyone trying to work out if Mikrotik has some equivalent of this SQM, looks like no. Well, not explicitly fq_codel or any extensions. Though using the sfq queue system with traffic shaping seems to give the same improvement to bufferbloat.
k__ 8y ago
Hm, I only get A and A+ results. But I'm on cabel, not DSL.
colin_fraizer 8y ago
"M-x desktop-clear" ?
philg_jr 8y ago
Anyone have any suggestions for pfSense? I played around with the traffic shaper, setting the scheduler type to CODELQ and limiting bandwidth to 95%, but it doesn't seem to do much from what I can tell while testing with the speedtest on dslreports.
- dtaht99 8y ago
  85-90% is a better starting point.
thermodynthrway 8y ago
If your network is all Linux hosts, configure BBR TCP congestion control and you won't ever have to worry about bufferbloat again (unless you use a lot of UDP) . There's a ton of research in this area, but to summarize, there's two main ways of controlling outgoing packet rate. Measuring RTT(round trip time) or packet loss rate. Unfortunately all the early (and still common) TCP protocols control their send rate by measuring packet drop. Sending faster and faster until upstream routers somewhere start dropping traffic. This has the effect of completely filling the outbound buffers of the slowest link in the chain. To combat this, most bufferbloat fighting algorithms focus on dropping traffic before buffers fill, namely RED (random early drop) and CODEL. Newer algorithms like TCP Vegas and BBR measure RTT and lower transmit rate when they detect buffers down the line filling, preventing bloat. In most cases, you still need a router configured to prevent bufferbloat because even a single naughty protocol on your network can fill outgoing buffers. The most important thing to know about controlling bufferbloat with QoS is that you MUST have an accurate estimation of your max upload/download rate. This is because you can only control bloat if you are in control of the slowest link where it will build first. All effective bufferbloat solutions rely on artificially making your router the slowest link, usually by limiting bandwitdh through them to ~90% actual. Once you have control of the slowest link you can pick which packets get dropped as the buffers fill. And using something like FQ_CODEL you can assign equal bandwitdh to all IP's on the network. The nice thing about controlling bandwidth this way vs hard speed limits per user is that it allows users to use as much bandwith as they want until staying below line rate requires sharing Also notice I keep saying outbound. You actually have far less control of bufferbloat on the inbound end. Best you can hope for is that dropping inbound packets coming in over the configured rate will stop whatever server from sending you more as quickly, but this isn't always the case. Luckily, in my experience, the vast majority of "lag" is due to outbound buffers filling, not inbound. So setting up bufferbloat fighting bandwidth sharing is usually extremely effective
- dtaht99 8y ago
  some comments: 1) BBR is currently not something I'd recommend at home. 2) The hope has always been that the core two bufferbloat-fighting algorithms (BQL, and fq_codel) would end up in the cable, fiber or dsl modem hardware, so that no shaping would be required, as there would be sufficient backpressure from the link itself to regulate the link intelligently. The cpu costs on this are nearly 0! BQL is 6 lines of new code in the device driver. fq_codel has been shipping in linux for 6 years. It's just a matter of turning it on... But: lacking that support from the ISP-supplied gear, we shape with htb + fq_codel, (as you say), to ~90% of the link rate... with another box - or even in the same box if the device driver can't be fixed and is overbuffered. We are painfully aware of how much cpu shaping costs but modern cpus usually have enough oomph to handle it. btw: We've come up with a new deficit shaper (sch_cake) that lets us get to ~100% of the isp bandwidth (so long as you get the wire framing exactly right), while providing vastly better queue management in the sqm system. 4) fq_codel is fair to flows, not devices. This works well in the general case, but has edge cases where abusive apps that open a lot of flows gain priority. Adding per host fq (while retaining per flow fq), even through nat, was the number 1 request from the users for sch_cake, and one of the main reasons why cake exists.
petermcneeley 8y ago
How is priority distinguished between two TCP streams? I thought that QoS basically didnt exist at the internet level of the network.
- dtaht99 8y ago
  Priority is the wrong way to think about it. Given all the sources of bursts on the internet today, fair queuing (or "flow queuing") has become the way to turn flows back into packets. there's an awful lot of lit on FQ, what we do with fq_codel is to not only interleave packets better but apply congestion control signals at the right time so competing tcp flows don't overwhelm the link (with under 10ms of buffering (v seconds common on fifo ISP links)). https://en.wikipedia.org/wiki/Fair_queuing https://en.wikipedia.org/wiki/Fair_queuing Of course, being perfectly fair to flows is sometimes undesirable, but making something strictly higher priority[1] is fraught with peril as you end up with a classification nightmare. Having fq gives you the best shot at smaller flows completing sooner, and of big flows sharing better with each other. Having vastly reduced buffering improves the responsiveness of competing TCP flows a lot, grabbing bandwidth whenever available, faster. My take on folk that want "prioritization" is ask them to try some variant of sqm with just fq and codel and get back to us. being fair with well managed buffers works really well.[2] [1] making something lower priority than best effort is actually a good idea. [2] but if you really want some flows or devices prioritized, see the sch_cake work mentioned on this thread. I still tend to think per host FQ is what many want rather than attempting to raise the priority of certain flows from certain services. https://arxiv.org/abs/1804.07617 https://arxiv.org/abs/1804.07617
  - petermcneeley 8y ago
    I have been away from this field from some time. The Cake seems to solve the problem with machines but not the problem with the same machine having streams with different latency/bw requirements. The priority way of solving this problem is round robin on packet priority and priority limit for machine/user (different queues per discritized priority level meaning no bufferbloat for high priority). The primary issue with this solution is that it would require the packets to be labeled.
ra1n85 8y ago
>Twiddling with QoS might help, but a faster internet connection probably won’t help at all. The key issue here is contention, or a busy egress interface - that's when buffering occurs. I do not understand why adding capacity, or reducing the probability of a busy egress interface, "won't help at all".
- tfha 8y ago
  The torrenting app will just consume all the extra space, leaving just as little for gaming as there was before.
  - dtaht99 8y ago
    totally untrue. fq_codel manages normal torrents just fine in the presence of gamer-style traffic. We tested against torrents in fixing bufferbloat a lot! Pure aqm systems like pie (in docsis 3.1) do pretty well also. cake (per host fq) can make even insane amounts of torrenting (or slashdotting!) bearable.
    - tgb 8y ago
      I think you misunderstood the above poster who was talking about the case without SQM (or I'm misunderstanding things because I've never heard any of these terms before now).
berkeleyjunk 8y ago
If you are interested in the background for this work I would greatly recommend the following two articles from ACM queue BufferBloat: What's Wrong with the Internet? - discussion with Vint Cerf, Van Jacobson, Nick Weaver, and Jim Gettys https://queue.acm.org/detail.cfm?id=2076798 https://queue.acm.org/detail.cfm?id=2076798 Bufferbloat: Dark Buffers in the Internet - Jim Gettys & Kathleen Nichols https://queue.acm.org/detail.cfm?id=2071893 https://queue.acm.org/detail.cfm?id=2071893
- criddell 8y ago
  The Security Now podcast covered this as well: https://www.grc.com/sn/sn-345.htm https://www.grc.com/sn/sn-345.htm
- dtaht99 8y ago
  That's good basics, but it's all fixed now. Jim's latest blog post goes into exausting detail. https://gettys.wordpress.com/category/bufferbloat/ https://gettys.wordpress.com/category/bufferbloat/
- JdeBP 8y ago
  See also John Nagle on the subject: * https://news.ycombinator.com/item?id=17455276 https://news.ycombinator.com/item?id=17455276
mekhami 8y ago
Can someone explain what I can do about this to a networking/hardware layman?
- vanburen 8y ago
  This video gives an overview of setting up the EdgeRouter X: https://www.youtube.com/watch?v=o-g2P3R84dw https://www.youtube.com/watch?v=o-g2P3R84dw
dtaht99 8y ago
I am immensely cheered up by the progress reports contributed by so many on this thread. But, can I ask that if you grok it, go fix it for two friends? Go fix it for a local small business, a coffee shop, or a hotel. And ask your ISP to fix it in their default gear? There's probably over 2b routers without bufferbloat fixes installed, and only if we work together to get them deployed will the internet as a whole get better, more capable of handling web, games, videoconferencing, and other applications that demand consistent low latency. Thanks.
- doctorsher 8y ago
  Thanks for everything that you do, Dave. Your bufferbloat and WiFi work is excellent, and I'm excited for the airtime fairness work with Toke to make it into more 802.11 drivers than ath9k. Cheers.
  - dtaht99 8y ago
    I was essentially in a overwork-induced coma for the last 18 months. The work was carried forward by (many, many) others. I was happy to wake up a few weeks back, and find RFC8290 published, and sch_cake being readied for mainline, and multiple commercial products finally shipping what we'd worked on all these years.
amelius 8y ago
Curious: is it possible already in Linux to give the highest bandwidth to the currently active application? And will routers play along?
- supertrope 8y ago
  Careful, logically this means giving BitTorrent #1 priority when brought to the top.
tgb 8y ago
Wow the manual test is very convincing: running fast.com's speedtest while pinging google makes the time increase from <30ms to over 1000ms! It's taking 1.5 seconds to ping google? I had no idea this could be happening.
- dtaht99 8y ago
  It's been happening to everyone. Spread the word.
doctorsher 8y ago
In my experience, changing the default DNS servers, enabling sch_cake, and minimizing shared spectrum interference are the most significant improvements for a home WiFi connection. Can anyone think of an additional dimension for improvement, besides upgrading the link itself?
- wtallis 8y ago
  > Can anyone think of an additional dimension for improvement, besides upgrading the link itself? Depending on the hardware, you can upgrade the link itself with newer, smarter WiFi drivers. After pretty much solving the bufferbloat problem for wired connections, many of the same developers moved on to fixing WiFi, and some fruits of that effort are currently available in OpenWRT and LEDE.
  - doctorsher 8y ago
    By 'enabling sch_cake', I meant that I use sch_cake on my wireless router (which I flashed with LEDE). Sorry, I was a bit light on the details.
treis 8y ago
Holllllleeeeeyyyyy sheeeeet!!! I went from a buffer bloat of 1.2s UP and 0.6s DOWN to effectively 0. It's like I have a whole new internet connection. OP and everyone else involved gets a million internet points from me!
gwillen 8y ago
I have a home internet connection that is a resold AT&T U-Verse connection. I doubt any of these fixes are available to me -- the extremely user-hostile equipment provides essentially no user-configurable options, and I have been told it has no switched mode, not even a secret one. So there's no way I can introduce my own router hardware unless I want to be double-NATted. (Also, it has a broken caching DNS server, and forces that broken server into its DHCP responses; the server it returns is not user-configurable.) So what's an ordinary consumer to do?
dsiemon 8y ago
It really is amazing the difference that modern AQMs like FQ-CoDel make. This is a bit of self promotion but we leverage FQ-CoDel in our product (https://www.preseem.com https://www.preseem.com) to help ISPs provide a better experience to their subscribers. Our customers regularly pass along anecdotes from subscribers who are very happy that they can now do big downloads or heavy streaming without breaking interactive applications like gaming.

207 points by velmu 8y ago

Do you want to have jitter, or do you want loss? In reality I don't see buffer bloat on the internet adding jitter of more than a few milliseconds. I do see loss though or 20, 50, even 100ms. I'd rather have 50ms of jitter than 50ms of loss, but that's just my application.

deleted 8y ago
[deleted]
Arie 8y ago
Why would you have to choose either? fq_codel and cake use a tiny bit of packet loss to get a sending host to back off, for example to keep a large download flow within the limits of your home link. Other flows aren't affected. Bufferbloat regularly adds hundreds of ms on home internet connections, you can get an indication of your bufferbloat on http://www.dslreports.com/speedtest http://www.dslreports.com/speedtest
- isostatic 8y ago
  Not on my connection. I would rather my UDP packet arrive 30ms late than not arrive - especially on high latency links where I want to process the packet before a 300ms round trip nack/retransmit has a chance to work. I don't see any buffer bloat or excessive jitter on my home internet (at least on wired connection) on BT ftth.
  - tgb 8y ago
    Note that the bufferbloat choice for me is whether or not I want my packets to arrive 1.5 seconds late, not 30ms.
- doctorsher 8y ago
  Anecdotally, I've seen larger amounts of packet loss and jitter when TCP accelerates faster into a loss event. The small amount of preventative loss reduces both of these values.
dtaht99 8y ago
there is no such thing as 50ms of loss. Loss, when your queue is short, is recovered from in an round trip. Bloat on the internet can add seconds of jitter. ( http://www.dslreports.com/speedtest/results/bufferbloat http://www.dslreports.com/speedtest/results/bufferbloat) But, I too am allergic to loss. fq_codel supports ECN, (explicit congestion notification), which is enabled, now, universally by IOS. As near as I can tell, the ECN usage of 6% (https://www.ietf.org/proceedings/98/slides/slides-98-maprg-tcp-ecn-experience-with-enabling-ecn-on-the-internet-padma-bhooma-00.pdf https://www.ietf.org/proceedings/98/slides/slides-98-maprg-t... ) in france is almost entirely from free.fr's deployment of fq_codel which they enabled by default in 2012 (!!!!!!). I had expected all the ISPs to have lept on this by now....
- isostatic 8y ago
  Yes there is. I have multiple UDP/RTP streams at different bitrates, from the same IP to the same IP. I then get a loss on a 30mbit stream (3000 packets a second) of 150 packets. At the exact same time I get a loss on a 20mbit stream of 100 packets, and a 10mbit stream of 50 packets. This is an outage for 150ms, probably because of a reroute in an MPLS network somewhere. My packets have already been emmitted by the time any round trip resend would have come back.
chacham15 8y ago
Tcp needs packets to be dropped quickly to get the feedback that your link can't handle the speed. Without that, it will appear that the speed of the network is fluctuating (as the buffer fills it appears your network is faster than it is, when the buffer is full it will start dropping packets which causes tcp to back off which causes the buffer to drain, now if tcp starts sending data before the buffer is empty you'll get a different apparent speed to tcp) and it is that fluctuation which causes the issues.

theogravity 8y ago

If you use Opnsense, I've written a tutorial on how you can configure it to reduce bufferbloat. https://forum.opnsense.org/index.php?topic=7423.0 https://forum.opnsense.org/index.php?topic=7423.0

vanburen 8y ago

I recently bought an Ubiquiti EdgeRouter X [1]. Which has a smart queue function that utilises fq_codel. Results from the DSLReports speed test [2], using the same router, the only difference being turning smart queue on: Smart queue off: https://i.imgur.com/zeY4rTd.png https://i.imgur.com/zeY4rTd.png Smart queue on: https://i.imgur.com/jfHpiFb.png https://i.imgur.com/jfHpiFb.png Notice the improvement in bufferbloat score from D to A+. Subjectively, I have noticed the connection seems more responsive and there no longer seem to be latency spikes when utilising all the upload bandwidth. It’s certainly worth considering bufferbloat, if you suffer from latency spikes when using all your upload bandwidth (I used to suffer from this a lot more, when I had an ADSL connection, with only 1 megabit upload). [1] https://www.ubnt.com/edgemax/edgerouter-x/ https://www.ubnt.com/edgemax/edgerouter-x/ [2] http://www.dslreports.com/speedtest http://www.dslreports.com/speedtest

Arie 8y ago
I love the ER-X because it's cheap, generally available and a fine router with SQM. Just keep in mind that with smart queue enabled it will top out at about 150-170Mbit. If you're on a faster connection the Edgerouter ER4 with its faster quad core CPU should be able to handle up to about 300-350Mbit. When you need to shape even more bandwidth, routers based on the Marvel XP Armada chipset do very well. I flashed a Linksys WRT1900ACS with OpenWRT and was able to shape about 600-750Mbps before it ran out of horsepower.
- matwood 8y ago
  > I love the ER-X because it's cheap I love my ER-X, but once I got my gigabit connection it can't quite keep up compared to plugging directly into the modem. It's close enough that I'm not looking to replace it, but the next time I need a router I may go the NUC build your own route.
  - 0xffff2 8y ago
    How does one go about building a router from a NUC? Don't you need two NIC's for a router? (One for the modem, one to connect a switch for the LAN.) Or have enough people given up on wired networking that they just build routers where the entire LAN is on WiFi?

Arie 8y ago

An awful lot of home broadband connections suffer from bufferbloat and even for the ones that don't, a single host can easily hog all the bandwidth. If you're used to getting lag in your VoIP or gaming when a house mate starts a stream/download/torrent, this can be fixed :) The cake traffic shaper in OpenWRT is amazing for fighting bufferbloat in your home network and it can also do almost perfect fairness in dividing the available bandwidth per LAN host with very little configuration. Just get it as part of the SQM tools in OpenWRT and enable it. For the per-host-fairness take a look at the "Make cake sing and dance" from this link: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details https://openwrt.org/docs/guide-user/network/traffic-shaping/... If you use an Edgerouter, you can get the cake traffic shaper but you'll have to do without the easy web interface OpenWRT has: https://community.ubnt.com/t5/EdgeRouter/Cake-and-FQ-PIE-compiled-for-the-EdgeRouter-devices/td-p/1679844 https://community.ubnt.com/t5/EdgeRouter/Cake-and-FQ-PIE-com...

graystevens 8y ago
Thanks for the Edgerouter link, saved me doing some Googling - Off to got and apply this now and see what difference this makes to the Bufferbloat tests. As Arie mentions, this is a little more involved on the EdgeOS stuff, but doesn't look too complex for those that are used to a CLI or two. Edit: Entirely off-topic, but an equally interesting find – There is a WireGuard client for the EdgeRouter line: https://community.ubnt.com/t5/EdgeRouter/Release-WireGuard-for-EdgeRouter/td-p/1904764 https://community.ubnt.com/t5/EdgeRouter/Release-WireGuard-f...
- voltagex_ 8y ago
  Ah crap, I wonder if I've done the wrong thing going with a switch + Security Gateway instead of an EdgeRouter.
  - andrewaylett 8y ago
    You get integration with the Unifi UI and a much easier configuration experience, you lose only because what you've got is (probably) overkill for residential use. If you've got more than one AP then you start to win again because you can power all your APs from the switch. If you were happy with your setup before hearing about the EdgeRouter, I hope you'll still be happy with it now :).

dtaht99 8y ago

Recently I ranted at the ISPs and at the network neutrality people. fq_codel (RFC8290) is now nearly ubiquitous in as the default queuing mechanism in most linux distributions, and it is long past time more ISPs supplied it in the gear they give customers. http://blog.cerowrt.org/post/net_neutrality_customers/ http://blog.cerowrt.org/post/net_neutrality_customers/ sch_cake (derived from fq_codel with an integral shaper) is about to enter the Linux mainline after years of baking in openwrt. https://lwn.net/SubscriberLink/758353/05c20f25115c852d/ https://lwn.net/SubscriberLink/758353/05c20f25115c852d/ We can end bufferbloat, now, on everything.

pja 8y ago
I enabled sch_cake on my router (an R7800 with Openwrt 18.06 on it) recently. Worked like a charm. Would happily recommend. I imagine the main thing that's holding back ISPs from using it on their consumer routers is the (minimal) configuration the user has to do to set to upload bandwidth limit. If ISP routers were also using decent queuing algorithms and not buffering like crazy then that wouldn't be necessary of course, but it doesn't seem like that's going to change any time soon.
- dtaht99 8y ago
  Well, many isp-born devices are provided with that info by the ISP at connect time, so the hope has always been that during configuration they'd just pass (for example) "bandwidth 10mbit docsis" to the network setup routine. DSL modems also typically get this info at startup.
philjohn 8y ago
The problem is that ISP provided kit is built to a price, so will tend to have a more aenemic processor. Shaping high bandwidth needs CPU cycles, or better yet, hardware offloading. Funnily enough, I've just got a firmware applied to my cable connection with Virgin Media in the UK that mitigates the Puma 6 issue on their provided DOCSIS 3 modem - it's eliminated buffer bloat as well, which is a nice side effect. People are listening, Docsis 3.1 includes active queue management for instance, which goes a long way to preventing the issue.

snikeris 8y ago

News to me. dslreports reported a C for bufferbloat. My router (eero) has a "labs" section that had SQM disabled. After enabling, dslreports is reporting an A.

stuaxo 8y ago

Is there a list of routers and their support for the different queuing methods (and how they are setup by default) ?

dtaht99 8y ago
DOCSIS 3.1 devices mandate pie, which helps a lot, but it's not as good as fq_codel, nor do they do shaping from the isp, which is kind of needed for all the cable links in the USA I've tried. Get one if you can, though, they are better across the board in many other ways. pfsense has fq_codel. Anything (1000s of routers) from lede/openwrt has the most advanced bufferbloat-fighting stuff in it, followed by dd-wrt, tomato, etc. If you need high bandwidths the multi-core arms are the best. fq_codel is pre-configured on all links (ethernet/usb/fiber/wifi/whatever) but if you need shaping to the ISP provided rate you need to configure it. All the research that went into fixing bufferbloat queuing problems everywhere landed in openwrt and lede first. Most of the research that improved tcp everywhere came out of google. Most gaming routers now sold commercially have some variant of fq_codel in them in their trade name ISP "qos" system. Also fq_codel derived anti-bufferbloat work has landed in many commercial wifi routers on the wifi side (eero, google wifi, some ubnt products, meraki, many others). The paper behind all that was: https://arxiv.org/pdf/1703.00064.pdf https://arxiv.org/pdf/1703.00064.pdf - happily that work was "good enough" to enable by default, and boy, does it make a difference if wifi is your bottleneck. The current premier dsl router with cake is evenroute. I think there are several new models from several manufacturers that are going to get it right, soon. Not tracking FIOS (gpon fiber) closely at the moment. Yes, fiber networks have bufferbloat, but it's harder to hit, and generally smaller than on dsl and cable technologies. I configured cake on sonic fiber recently and got 60ms back. (going from 60ms latency under load to 3ms ) Regrettably shaper setup is finicky and requires a few minutes of testing with a site like dslreports or a tool like flent.org to get right. If more ISPs published their shapers' bitrate and burst rate settings, life would be easier here... but the hope has always been they'd just ship a router with this stuff on and remotely configured to be "right". In some ways, what are you doing about bufferbloat is
- dtaht99 8y ago
  Let's see. The turris omnia is a very good router (but only available in europe). For oomph (gbit shaping) people often leverage lede on a pcengines apu2 or run a full distro of pfsense or linux on it.

josteink 8y ago

As someone who has meddled with QOS in the past, this page didn't really feel very concrete to me. Especially regarding SQM... Is SQM just a new name for QOS or am I missing something?

rusk 8y ago
Isn't QoS a more general qualitative, perceived phenomenon? SQM sounds like it relates to something you'd do to improve QoS but then only at one link in the chain, at one layer.
- dtaht99 8y ago
  I wish! QoS could have been a good term to keep using if the existing deployments of it on the Internet it wasn't hopelessly mapped generally to mere packet prioritization (diffserv) which doesn't actually work on today's internet. QoE is a better, less overloaded, in the "qualitative, perceived" sort of description. So (after endless debates) - we defined SQM as as a superset of classic as-defined-for-internet QoS and hoped for the best. (https://www.bufferbloat.net/projects/cerowrt/wiki/Smart_Queue_Management/ https://www.bufferbloat.net/projects/cerowrt/wiki/Smart_Queu... ) There's been plethora of other trade names for what we do with htb+fq_codel - streamboost, adaptive qos, etc. I like that eero, edgerouter, and openwrt and derivatives also call what we do sqm. It simplifies the discussion, and the core scripts for linux generically are available as the sqm-scripts on github. I also hope we see more RFPs specifying RFC8290.
  - rusk 8y ago
    > deployments of it on the Internet it wasn't hopelessly mapped generally to mere packet prioritization (diffserv) Sorry yeah, I forgot about how the term has been mangled over the years. I was coming from a 3GPP perspective where the term is specifically defined as experience [0] In particular: * only the QoS perceived by end-user matter * QoS definitions have to be future proof; * QoS has to be provided end-to-end. * QoS attributes (or mapping of them) should not be restricted to one or few external QoS control mechanisms Not the first time the ivory tower of Telecomms has been out of touch with the outside world ... [0] 4.1 (p. 7) https://www.etsi.org/deliver/etsi_ts/123100_123199/123107/05.04.00_60/ts_123107v050400p.pdf https://www.etsi.org/deliver/etsi_ts/123100_123199/123107/05...

magoon 8y ago

I cured bhfferbloat with a cheap DD-WRT capable router (NOT Puma 6 chipset). I found it important to limit the WAN bandwidth on my router to about 15% below the max speed I’m paying for. What a difference! Netgear R6300V2 Using fq_codel because the others use more CPU than I’m confident these devices can handle. I find that most people suffer from bufferbloat unknowingly, having bought fancy routers 5-10 years ago.

scruffyherder 8y ago

Residential internet in Hong Kong can go as high as 10gig. I don't see anything about links anywhere around 10gig or higher, if I wanted this on my 100gig backbone or 40gig metro links.

dtaht99 8y ago
Bufferbloat happens on high speed links like those but amount of bloat you see is in the 30ms - 60ms range (vs seconds(!!) on home links). Bloat happens mostly (aside from microbursts) on overloaded links - and high speed backbones are typically overprovisioned so the problem only shows up when there's an outage or fiber cut. Example of what happens today on a fiber cut: http://blog.cerowrt.org/post/bufferbloat_on_the_backbone/ http://blog.cerowrt.org/post/bufferbloat_on_the_backbone/ IF things like fq_codel were deployed on those we'd not see latencies climb that much at all, we'd see bandwidths decline to the actual capacity available - and only the biggest flows would be hit to do so. FQ_codel is lightweight enough to fit directly in high speed hardware and it does indeed run on 40gigE plus devices on linux, ddpk, BSD, in software. But it takes a long time for new chipsets to adopt new algorithms even if they incorporate support for deeply desirable features like ECN. That said... 10Gbit to the home.... ooohhhhh. it's really hard to bloat that!

meta_AU 8y ago

Anyone trying to work out if Mikrotik has some equivalent of this SQM, looks like no. Well, not explicitly fq_codel or any extensions. Though using the sfq queue system with traffic shaping seems to give the same improvement to bufferbloat.

k__ 8y ago

Hm, I only get A and A+ results. But I'm on cabel, not DSL.

colin_fraizer 8y ago

"M-x desktop-clear" ?

philg_jr 8y ago

Anyone have any suggestions for pfSense? I played around with the traffic shaper, setting the scheduler type to CODELQ and limiting bandwidth to 95%, but it doesn't seem to do much from what I can tell while testing with the speedtest on dslreports.

dtaht99 8y ago
85-90% is a better starting point.

thermodynthrway 8y ago

If your network is all Linux hosts, configure BBR TCP congestion control and you won't ever have to worry about bufferbloat again (unless you use a lot of UDP) . There's a ton of research in this area, but to summarize, there's two main ways of controlling outgoing packet rate. Measuring RTT(round trip time) or packet loss rate. Unfortunately all the early (and still common) TCP protocols control their send rate by measuring packet drop. Sending faster and faster until upstream routers somewhere start dropping traffic. This has the effect of completely filling the outbound buffers of the slowest link in the chain. To combat this, most bufferbloat fighting algorithms focus on dropping traffic before buffers fill, namely RED (random early drop) and CODEL. Newer algorithms like TCP Vegas and BBR measure RTT and lower transmit rate when they detect buffers down the line filling, preventing bloat. In most cases, you still need a router configured to prevent bufferbloat because even a single naughty protocol on your network can fill outgoing buffers. The most important thing to know about controlling bufferbloat with QoS is that you MUST have an accurate estimation of your max upload/download rate. This is because you can only control bloat if you are in control of the slowest link where it will build first. All effective bufferbloat solutions rely on artificially making your router the slowest link, usually by limiting bandwitdh through them to ~90% actual. Once you have control of the slowest link you can pick which packets get dropped as the buffers fill. And using something like FQ_CODEL you can assign equal bandwitdh to all IP's on the network. The nice thing about controlling bandwidth this way vs hard speed limits per user is that it allows users to use as much bandwith as they want until staying below line rate requires sharing Also notice I keep saying outbound. You actually have far less control of bufferbloat on the inbound end. Best you can hope for is that dropping inbound packets coming in over the configured rate will stop whatever server from sending you more as quickly, but this isn't always the case. Luckily, in my experience, the vast majority of "lag" is due to outbound buffers filling, not inbound. So setting up bufferbloat fighting bandwidth sharing is usually extremely effective

dtaht99 8y ago
some comments: 1) BBR is currently not something I'd recommend at home. 2) The hope has always been that the core two bufferbloat-fighting algorithms (BQL, and fq_codel) would end up in the cable, fiber or dsl modem hardware, so that no shaping would be required, as there would be sufficient backpressure from the link itself to regulate the link intelligently. The cpu costs on this are nearly 0! BQL is 6 lines of new code in the device driver. fq_codel has been shipping in linux for 6 years. It's just a matter of turning it on... But: lacking that support from the ISP-supplied gear, we shape with htb + fq_codel, (as you say), to ~90% of the link rate... with another box - or even in the same box if the device driver can't be fixed and is overbuffered. We are painfully aware of how much cpu shaping costs but modern cpus usually have enough oomph to handle it. btw: We've come up with a new deficit shaper (sch_cake) that lets us get to ~100% of the isp bandwidth (so long as you get the wire framing exactly right), while providing vastly better queue management in the sqm system. 4) fq_codel is fair to flows, not devices. This works well in the general case, but has edge cases where abusive apps that open a lot of flows gain priority. Adding per host fq (while retaining per flow fq), even through nat, was the number 1 request from the users for sch_cake, and one of the main reasons why cake exists.

petermcneeley 8y ago

How is priority distinguished between two TCP streams? I thought that QoS basically didnt exist at the internet level of the network.

dtaht99 8y ago
Priority is the wrong way to think about it. Given all the sources of bursts on the internet today, fair queuing (or "flow queuing") has become the way to turn flows back into packets. there's an awful lot of lit on FQ, what we do with fq_codel is to not only interleave packets better but apply congestion control signals at the right time so competing tcp flows don't overwhelm the link (with under 10ms of buffering (v seconds common on fifo ISP links)). https://en.wikipedia.org/wiki/Fair_queuing https://en.wikipedia.org/wiki/Fair_queuing Of course, being perfectly fair to flows is sometimes undesirable, but making something strictly higher priority[1] is fraught with peril as you end up with a classification nightmare. Having fq gives you the best shot at smaller flows completing sooner, and of big flows sharing better with each other. Having vastly reduced buffering improves the responsiveness of competing TCP flows a lot, grabbing bandwidth whenever available, faster. My take on folk that want "prioritization" is ask them to try some variant of sqm with just fq and codel and get back to us. being fair with well managed buffers works really well.[2] [1] making something lower priority than best effort is actually a good idea. [2] but if you really want some flows or devices prioritized, see the sch_cake work mentioned on this thread. I still tend to think per host FQ is what many want rather than attempting to raise the priority of certain flows from certain services. https://arxiv.org/abs/1804.07617 https://arxiv.org/abs/1804.07617
- petermcneeley 8y ago
  I have been away from this field from some time. The Cake seems to solve the problem with machines but not the problem with the same machine having streams with different latency/bw requirements. The priority way of solving this problem is round robin on packet priority and priority limit for machine/user (different queues per discritized priority level meaning no bufferbloat for high priority). The primary issue with this solution is that it would require the packets to be labeled.

ra1n85 8y ago

>Twiddling with QoS might help, but a faster internet connection probably won’t help at all. The key issue here is contention, or a busy egress interface - that's when buffering occurs. I do not understand why adding capacity, or reducing the probability of a busy egress interface, "won't help at all".

tfha 8y ago
The torrenting app will just consume all the extra space, leaving just as little for gaming as there was before.
- dtaht99 8y ago
  totally untrue. fq_codel manages normal torrents just fine in the presence of gamer-style traffic. We tested against torrents in fixing bufferbloat a lot! Pure aqm systems like pie (in docsis 3.1) do pretty well also. cake (per host fq) can make even insane amounts of torrenting (or slashdotting!) bearable.
  - tgb 8y ago
    I think you misunderstood the above poster who was talking about the case without SQM (or I'm misunderstanding things because I've never heard any of these terms before now).

berkeleyjunk 8y ago

If you are interested in the background for this work I would greatly recommend the following two articles from ACM queue BufferBloat: What's Wrong with the Internet? - discussion with Vint Cerf, Van Jacobson, Nick Weaver, and Jim Gettys https://queue.acm.org/detail.cfm?id=2076798 https://queue.acm.org/detail.cfm?id=2076798 Bufferbloat: Dark Buffers in the Internet - Jim Gettys & Kathleen Nichols https://queue.acm.org/detail.cfm?id=2071893 https://queue.acm.org/detail.cfm?id=2071893

criddell 8y ago
The Security Now podcast covered this as well: https://www.grc.com/sn/sn-345.htm https://www.grc.com/sn/sn-345.htm
dtaht99 8y ago
That's good basics, but it's all fixed now. Jim's latest blog post goes into exausting detail. https://gettys.wordpress.com/category/bufferbloat/ https://gettys.wordpress.com/category/bufferbloat/
JdeBP 8y ago
See also John Nagle on the subject: * https://news.ycombinator.com/item?id=17455276 https://news.ycombinator.com/item?id=17455276

mekhami 8y ago

Can someone explain what I can do about this to a networking/hardware layman?

vanburen 8y ago
This video gives an overview of setting up the EdgeRouter X: https://www.youtube.com/watch?v=o-g2P3R84dw https://www.youtube.com/watch?v=o-g2P3R84dw

dtaht99 8y ago

I am immensely cheered up by the progress reports contributed by so many on this thread. But, can I ask that if you grok it, go fix it for two friends? Go fix it for a local small business, a coffee shop, or a hotel. And ask your ISP to fix it in their default gear? There's probably over 2b routers without bufferbloat fixes installed, and only if we work together to get them deployed will the internet as a whole get better, more capable of handling web, games, videoconferencing, and other applications that demand consistent low latency. Thanks.

doctorsher 8y ago
Thanks for everything that you do, Dave. Your bufferbloat and WiFi work is excellent, and I'm excited for the airtime fairness work with Toke to make it into more 802.11 drivers than ath9k. Cheers.
- dtaht99 8y ago
  I was essentially in a overwork-induced coma for the last 18 months. The work was carried forward by (many, many) others. I was happy to wake up a few weeks back, and find RFC8290 published, and sch_cake being readied for mainline, and multiple commercial products finally shipping what we'd worked on all these years.

amelius 8y ago

Curious: is it possible already in Linux to give the highest bandwidth to the currently active application? And will routers play along?

supertrope 8y ago
Careful, logically this means giving BitTorrent #1 priority when brought to the top.

tgb 8y ago

Wow the manual test is very convincing: running fast.com's speedtest while pinging google makes the time increase from <30ms to over 1000ms! It's taking 1.5 seconds to ping google? I had no idea this could be happening.

dtaht99 8y ago
It's been happening to everyone. Spread the word.

doctorsher 8y ago

In my experience, changing the default DNS servers, enabling sch_cake, and minimizing shared spectrum interference are the most significant improvements for a home WiFi connection. Can anyone think of an additional dimension for improvement, besides upgrading the link itself?

wtallis 8y ago
> Can anyone think of an additional dimension for improvement, besides upgrading the link itself? Depending on the hardware, you can upgrade the link itself with newer, smarter WiFi drivers. After pretty much solving the bufferbloat problem for wired connections, many of the same developers moved on to fixing WiFi, and some fruits of that effort are currently available in OpenWRT and LEDE.
- doctorsher 8y ago
  By 'enabling sch_cake', I meant that I use sch_cake on my wireless router (which I flashed with LEDE). Sorry, I was a bit light on the details.

treis 8y ago

Holllllleeeeeyyyyy sheeeeet!!! I went from a buffer bloat of 1.2s UP and 0.6s DOWN to effectively 0. It's like I have a whole new internet connection. OP and everyone else involved gets a million internet points from me!

gwillen 8y ago

I have a home internet connection that is a resold AT&T U-Verse connection. I doubt any of these fixes are available to me -- the extremely user-hostile equipment provides essentially no user-configurable options, and I have been told it has no switched mode, not even a secret one. So there's no way I can introduce my own router hardware unless I want to be double-NATted. (Also, it has a broken caching DNS server, and forces that broken server into its DHCP responses; the server it returns is not user-configurable.) So what's an ordinary consumer to do?

dsiemon 8y ago

It really is amazing the difference that modern AQMs like FQ-CoDel make. This is a bit of self promotion but we leverage FQ-CoDel in our product (https://www.preseem.com https://www.preseem.com) to help ISPs provide a better experience to their subscribers. Our customers regularly pass along anecdotes from subscribers who are very happy that they can now do big downloads or heavy streaming without breaking interactive applications like gaming.