8 ms·
Linux QoS for humans
- tra3 11y agoThis looks intriguing, thank you! Over the last 10 years I've tried a number of things to get QoS working on Linux, with not much success. My most successful (and recent) attempt was to purchase a Mikrotik, but to be honest I'm not totally satisfied. This seems like it might be a better solution!
- ktsaou 11y agothanks!
- pcunite 11y agoWhat trouble are you having with MikroTik? I enjoy them.
- tra3 10y agoMikrotik has been rock solid for me. It's my lack of knowledge really. I adapted someone's QoS configuration and it's been working really well. Much better than anything I've tried before. My only complaint is the fact is the proprietary nature of Mikrotik. When I was running a small linux box as a router, or when I was running DD-WRT I could always extend the functionality in the ways that's not possible with Mikrotik. The configuration, QoS, everything is Mikrotik specific. I understand that it's reminiscent of enterprise grade hardware though.
- MattSteelblade 11y agoProject looks extremely interesting. Will have to try it out.
- ktsaou 11y agothanks!
- bcook 11y agoQoS can help with DDoS if the DDoS is L7, but most DDoS attacks are bandwidth exhaustion attacks, which requires upstream mitigation. I completely agree that "tc" is one of the most confusing CLI programs I have encountered. tc & iptables pushed me away from Linux (for router purposes) to *BSD and pf has been a refreshing experience.
- ktsaou 11y agoYour are so right! However most of the DDoS attacks I have experienced were amateur, L7 (wordpress, POSTs, etc).
- jrcii 11y agoI've really enjoyed pf on OpenBSD over the years. My only frustration has been the ever-changing backwards-incompatible rule syntax which perpetually makes large swaths of documentation useless, including the most recent printed book on pf and numerous online sources.
- geggam 11y agoman pf.conf .. always works :)
- txutxu 11y agoCome on, it's not so hard. You mark the packages in the mangle table and you distribute the defined bw with tc. OK, I confess... all my life I've reused a tc script I inherited from a gentoo sysadmin. It works, I edit the available bandwitch and magic happens. The iptables cmd is more easy to understand and I even did an output parser and input generators, but tc is hard.
- geggam 11y agoGuess this is too old school ? http://lartc.org/wondershaper/ http://lartc.org/wondershaper/
- bcook 11y agoYeah, it's even deprecated by people like Dave Taht (fq_CoDel). http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershape...
- tra3 11y agoI'm curious -- Have you ever tried it? I could never get it to work.
- ktsaou 11y agoI never managed to make it work either.
- eosrei 11y agoIt works well for me. I run it on my local machine to make the network more responsive during large downloads. sudo wondershaper eth0 7500 850
- wtallis 11y agoYou'll get better results from sysctl -w net.core.default_qdisc=fq_codel on as many machines as possible, and use something like OpenWRT's SQM instead of wondershaper on your router.
- homarp 11y agopreviously https://news.ycombinator.com/item?id=11477915 https://news.ycombinator.com/item?id=11477915
- ktsaou 11y agoYou are right! The title (or the timing) did make a difference though.
- dfc 11y agoI hope your take away is not that it is a good idea to embellish in the titles. There is a reason editorializing in titles is frowned upon; it's a hyperbolic arms race and everyone is a loser.
- ktsaou 11y agoI am just trying to understand how this works. Probably is the timing. How do you explain the difference in involvement between the 2 posts? I would love to understand what made the difference. Can you explain it?
- dfc 11y agoI cant explain it. What I am concerned about is that you are focusing on it. Would you give your links linkbaity titles if it meant more traffic?
- dangerlibrary 11y agoEmpirically, yes.
- ktsaou 11y agoI am an open source developer. I don't sell anything. I am just trying to let people know there is something they can use to make their lives easier. I don't care about traffic (actually I don't want unrelated traffic - a lot of noise - the key resource wasted is my time), but I do care a lot about involvement. Since a title can actually trigger involvement, I understand it is good, especially for the readers. Isn't it? To my understanding, it is good since people that should have been informed about the context in question, have actually been found and informed. Check this thread for example. A lot of people found something new, learned something. Compare it to the previous one. No one learned anything. So, what do you think? Shall I pay some attention on titles and their effectiveness?
- mixmastamyk 11y agoHow does the QoS interact with containers? Would probably need to configure at the host level, correct? Wow, that "netdata" daemon/dashboard this is associated with is awesome. I'm wondering if it can be extended for more app metrics, like a custom solution we had at a former employer using grafana.
- ktsaou 11y agoRegarding QoS and containers, it can be applied at either side (veth is good too). Regarding netdata, of course it can be extended. Check this: https://github.com/firehol/netdata/wiki/External-Plugins https://github.com/firehol/netdata/wiki/External-Plugins A lot of other users are already writing their plugins - check the github issues
- vesinisa 11y agoI am personally using tcng, which is a configuration language that already makes tc much more usable. tcng on $ourceForge: http://tcng.sourceforge.net/ http://tcng.sourceforge.net/ Source code mirror by Debian (if wary of downloading from $ourceForge): https://packages.debian.org/source/wheezy/tcng https://packages.debian.org/source/wheezy/tcng
- ktsaou 11y agoInteresting. It seems however that its latest release is 2004 !
- vesinisa 11y agoNo problem, it still works. It just internally calls tc. It's a language that makes it easier to write tc classes and matching rules. The documentation is here: http://linux-ip.net/gl/tcng/ http://linux-ip.net/gl/tcng/ The QoS userspace interface (tc) has not changed much at all, and at least my configuration has no issues.
- candl 11y agoI am not that familiar with iptables/tc, but could the tc definitions created by fireqos be consumed by iptables + ipset (say I have an ipaet storing ip addresses and I want to provide a different qos profile for each ip) ?
- ktsaou 11y agoQoS and Firewall are different things in Linux. I have developed FireHOL (iptables, ipsets) and FireQOS (qos) to handle everything. They cooperate to share for example MARKs, but can also be used independently.
- mindslight 11y agoHow does FireQOS/FireHOL handle dynamic behavior like interfaces coming/going? My problem isn't so much the tc/ip/iptables syntax or semantics, but stateful mismatch like 'ip route' disappearing when an interface goes down, while 'ip rule' sticks around. Once a config gets that complex it's nice to keep in one place rather than scattered through various files. The easiest hacks are to let rules build up, or to reload everything on any ifup, but ugh. Also, manually allocating numeric table/mark/prio/class numbers is annoying.
- ktsaou 11y agoiptables (FireHOL) is generally autonomous and does not depend on interfaces being available at activation. tc (FireQOS) requires the interfaces to be up and running. To overcome these issues a) the configuration of both tools is a BASH script itself - do whatever you like in them: variables, conditions, loops, etc and b) in the distribution there is another tool called 'link-balancer' that applies inheritance to routing tables (!) and dynamically alters ip rule filters - it can also manage the balance of gateways.
- ktsaou 11y agoAnd all these tools (FireHOL, FireQOS, link-balancer) are just BASH scripts themselves (big ones of course). FireHOL: https://github.com/firehol/firehol/blob/master/sbin/firehol.in https://github.com/firehol/firehol/blob/master/sbin/firehol.... FireQOS: https://github.com/firehol/firehol/blob/master/sbin/fireqos.in https://github.com/firehol/firehol/blob/master/sbin/fireqos.... Link-Balancer: https://github.com/firehol/firehol/blob/master/sbin/link-balancer.in https://github.com/firehol/firehol/blob/master/sbin/link-bal...
- throw7 11y agoIs it overly pedantic for me to say QoS has nothing to do with inbound traffic or is that an older, say, traditional definition? My understanding is one can only "shape" outbound traffic. You are under no control at all of the incoming packets.
- ktsaou 11y agoLinux kernel magic. It is called IFB. Check this explanation at the FireQOS tutorial: https://github.com/firehol/firehol/wiki/FireQOS-Tutorial#controlling-incoming-traffic https://github.com/firehol/firehol/wiki/FireQOS-Tutorial#con...
- brbsix 11y agoYou can install an ingress queue and apply a filter to inbound traffic. There are few other more arcane techniques but you're right that control is quite limited. Here's an example for something I was working on yesterday (traffic policing to simulate slow download): sudo tc qdisc add dev eth0 handle ffff: ingress sudo tc filter add dev etch0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 30kbit burst 3k drop flowid :1 Here's an example of using IFB to shape incoming traffic (what the OP was referencing): http://serverfault.com/a/386791/301799 http://serverfault.com/a/386791/301799
- wtallis 11y agoIt's definitely an outdated definition that comes from the mindset that QoS means shaping traffic according to what protocol it uses. That's far too narrow a meaning, and the technique of explicitly classifying traffic is mostly obsolete given the availability of AQMs and FQ that need little or no configuration.
- baghali 11y agoThis is awesome. Would be great if it gets OpenWRT integration as well.
- baghali 11y agoOops, never mind: http://firehol.org/installing/openwrt/ http://firehol.org/installing/openwrt/
- brbsix 11y agoThat first paragraph is so right. `tc` is as close to deep magic as anything I've come across in a Linux box.
- Animats 11y agoWhen I used to work on network congestion, I took the position that each stream is entitled to one packet in flight. After that, we can talk about throttling. That's why I invented fair queuing. TCP will throttle down nicely to one packet in flight, and the connection won't stall, although it may be slow. Routers which discard all the packets from a single stream are doing a very bad job, unless they're dealing with hostile action. Much of the trouble comes from network devices with big, dumb, FIFO buffers. DSL routers are notorious for this on their uplink side. So are Ethernet interfaces, which have a big FIFO buffer. It's important to avoid too much dumb buffering in the chain between TCP and the Ethernet cable/WiFi air link. A typical problem is ACKs from downloads being lost during heavy uplink usage. If you're a choke point (much more bandwidth in than out), big dumb FIFO buffers will suck. You at least need fair queuing. The "bufferbloat" crowd is into this. Linux QOS is more about fixing this as an endpoint. You're trying to avoid overloading the next choke point upstream. This works only if everybody using the next choke point upstream plays nice. One bad guy will ruin it for everybody.
- sargun 11y agoNagle, much of the issues you're talking about with bad middleboxes is about signaling or optics into their status. We've had a standard for some years to deal with this (ECN). Why don't you think ECN is deployed at all? It seems to me it would be beneficial for all parties.
- wtallis 11y agoECN isn't a solution so much as it is an optional side-effect of routers deploying AQM—the real solution. Widespread deployment of AQM depends on having an algorithm that is both highly effective and requires no tuning or configuration. ECN deployment further requires being confident that broken routers won't cause problems and making a decision of where to put the threshold for marking relative to the threshold for dropping. You want there to be an incentive for endpoints to negotiate ECN, but you don't want there to be too much incentive for endpoints to declare ECN capability and then not respond to marking.
- Animats 11y ago
- Scramblejams 11y agoAlso featured recently on HN is this nice tool: https://github.com/thombashi/tcconfig https://github.com/thombashi/tcconfig
- cbsmith 11y agoCongratulations on breaking the trend of "for humans" projects being written in Python. ;-)
- Klingner 10y agoI’d just like to add some other cool QoS-projects to the discussion so that maybe more people can find a project that suits their own needs. I’ve always been fond of QoS, mainly because it was a necessity with five family members and just one measly DSL-connection, second because it’s just neat to use QoS to prevent people from breaking your connection and making everything much more effective. 1. It seems like the first QoS-solution I ever used is still being actively developed. It’s called Mastershaper and you can find it here: https://oss.netshadow.net/redmine/projects/mastershaper https://oss.netshadow.net/redmine/projects/mastershaper Though it seems like the maintainer is doing some heavy work right now so there is no usable version at the moment. In my old days Mastershaper was programmed in PHP and it either worked with iptables as filter or with the faster tc-filters. It had a nice GUI and also some flow analysis through a graph. It was made to run under Debian and the like. 2. The other project I would like to mention is a router firmware called Tomato. The original website might be still available but the original author has ceased development for several years now. It’s only being developed by its community now. You can find the forum here: http://linksysinfo.org/index.php?forums/tomato-firmware.33/ http://linksysinfo.org/index.php?forums/tomato-firmware.33/ This firmware can be run on several open commercial home-routers (MIPSR1, MIPSR2, ARM), one of them being the old and very popular WRT54GL. But of course it also runs on more recent models. Tomato has several flavours. Toastman’s Tomato is known for its well implemented QoS and stability. You can find it here: http://www.linksysinfo.org/index.php?threads/toastman-mips-and-arm-releases.36106/ http://www.linksysinfo.org/index.php?threads/toastman-mips-a... Tomato probably has one of the most easily usable QoS-systems there is. Once installed, all you need to do is measure your line speed throughout the day, deduce a safety margin of about 15-30% to start with, enter your upstream and downstream values into the GUI, hit “Save” and you are running a pretty decently pre-configured QoS-system. It even supports overhead calculations for ADSL. Unfortunately it doesn’t work with IPv6, yet, and also uses the old netfilter L7-filters, which are pretty outdated. A switch to nDPI hasn’t been undertaken, yet. If you think about using Tomato with QoS, keep in mind that QoS needs a lot of CPU power, so it’s important to know your line speed and buy accordingly. 3. Another firmware that probably has well-made QoS is Gargoyle: https://www.gargoyle-router.com/ https://www.gargoyle-router.com/ I haven’t been using it but maybe it’s worth a look.