6 ms·

Linux's awesome and underused traffic control facilities

224 points by powertower 14y ago

powertower 14y ago
The little I've been able to find on the kernel and TC (Traffic Control) is here http://www.tldp.org/HOWTO/html_single/Traffic-Control-HOWTO/ http://www.tldp.org/HOWTO/html_single/Traffic-Control-HOWTO/
- joshbaptiste 14y ago
  indeed, The main reason tc(1) is underused is because it is under explained.
  - nknight 14y ago
    Even were it adequately explained, it would still be underused, being a pretty crappy interface. I find it virtually the antithesis of iptables, whose syntax and meaning is fairly intuitive and easy both to remember, and derive from a few example rules.
    - JoachimSchipper 14y ago
      I agree that tc is not a very good user interface, but I'm not sure I agree with you that IPTables is. Consider http://www.openbsd.org/faq/pf/example1.html#allrules http://www.openbsd.org/faq/pf/example1.html#allrules - a nice configuration language does look better than a shell script.
      - zobzu 14y ago
        Well I happen to disagree with that. So yeah, the script is very small. But why is it? Because aliases are used everywhere. Over simplifying things. When you read it, you don't know what it does. When you read the 3 pages of explanations, you get the idea, but you still don't know exactly what it does. At least with iptables you do. And you can match virtually anything (like a bit in a packet if you like) The real interface issue is TC.
      - nknight 14y ago
        Uh, yeah, no, that's not my idea of a good user interface. If anything, you've just convinced me never to touch pf.
        patrickgzill 14y ago
        Go ahead and touch it anyways. I can't imagine firewalling without it.
jen_h 14y ago
Have abused tc in the past to simulate high-latency remote dial-in connections while testing network equipment (netem delay ftw), but didn't realize that you could restrict by IP, thereby still being able to use the system for something other than dog-slow response. This is very cool - thanks for posting!
res0nat0r 14y ago
There really is a lack of good tc documentation out there and honestly interfacing with tc sucks. HSFC Scheduling with Linux http://linux-ip.net/articles/hfsc.en/ http://linux-ip.net/articles/hfsc.en/ is a good read.
fendale 14y ago
Quite a few years ago, I setup tc to do a bit of traffic shaping on my home network. Instead of having a standard router, I have a Linux box with two network cards using iptables to do NAT. I connect my cable modem to one of the network cards and then I connect my home network to the other. Way back, I found I had two problems - if I was doing something that did a lot of uploading (eg seeding a torrent), my download speed would tank because the tcp/ip packets were bogged down in the upload queue. I also like to be able to ssh into my machine while away from home, and if I did that while uploads were going on, the ssh terminal was basically unusable. Back then I discovered tc - it was able to make TCP control packets (ack's etc) and ssh related packets jump to the head of the queue, and the other upload stuff have a lower priority. Looking back at the script I created now, I don't really understand what it is doing, but it remember it worked. I still run it today and just take it for granted. I tested it by start a large upload while ssh'ed into the box from the internet - I turned off the tc rules and the terminal became unusably laggy, and then I turned the rules on and it worked perfectly.
- sneak 14y ago
  This helps a lot with that, as well: http://mosh.mit.edu/ http://mosh.mit.edu/
  - Ralith 14y ago
    How would that help at all?
    - asharp 14y ago
      From my understanding, I think they meant that mosh would deal with the more congested conditions better then ssh, thus providing a better experience. Just a hunch though. That being said, mosh is rather cool ^_^
- js2 14y ago
  Once upon a time, I used to use http://lartc.org/wondershaper/ http://lartc.org/wondershaper/ for that. I never found it worked very well for me though.
sunyc 14y ago
it is under used because most of the time, you just don't have to care. and when it matters it lack the depth you need, like layer 7 detection. it is almost always better to just pay for bigger pipes.
- rbanffy 14y ago
  Wouldn't layer 7 detection drive CPU usage up for a somewhat different use case? You are not trying to give video traffic a lower priority - you are trying to prevent things like backups clogging your pipes.
lobster_johnson 14y ago
We use this with our office SDSL connection to prioritize normal traffic and deprioritize things like backups (which used to steal all the bandwidth and drive pings up to 1+ seconds). Works swell, highly recommended.
jcurbo 14y ago
tc is very useful for managing traffic over low bandwidth connections. I have used it to control traffic over satellite links (T1 scale throughput) with great success.
RobAtticus 14y ago
I've been using tc a lot recently in the research I've been doing and I have to agree that finding good documentation/examples is somewhat difficult to do. There are a lot of weird gotchas (the naming between uplink and downlink being one of them -- root vs ingress). Of course, I'm working on a better framework for controlling network resources, so I have to learn how tc works to provide a simpler interface for doing the things it does.
- jvehent 14y ago
  Simpler is better only if it provides the same level of flexibility. the TC command line is hard to learn at first, because it is extremely flexible... and after all, that's what you would expect from a packet scheduler.
KonradKlause 14y ago
Why underused? It's it well known and heavily used.
MatthewIfe 14y ago
As the author of the original answer, I have to agree that the interface is shambolic compared to iptables. I think much of this is due to using the same interface ideas as what is in the "ip" command which follows the same principles and is also underused in favour of ifconfig. They are actually part of the same package in fact. ifconfig has technically been deprecated in favour of ip for about 8 years. In any case, resource control on Linux is very very good these days. From traffic control to control groups (you can also manage tc classes in control groups too), its worth any system administrator having things like tc and cgroups in their arsenal so that the operating system controls the applications / users and not the other way around. A pain in the ass compared to other CLI interfaces I agree, but I personally place resource management high on my things to know about, in the same way I put access management and storage management there too.
- secure 14y ago
  I use 'ip' for all my network configuration/debugging and I think it’s a great tool. I can’t say the same for tc, which I tried to get into multiple times and failed. I will try it once more with your good explanation and understandable example. Thanks!
jvehent 14y ago
I wrote this: Journey to the Center of the Linux Kernel: Traffic Control, Shaping and QoS http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:networking:traffic_control http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:n...
herf 14y ago
I've used tc on my openwrt (Linksys!) firewall for years. A network genius friend helped set it up. Why are there no consumer firewalls with great defaults for this? How fantastic in the era of cloud storage and network backup to have long-running uploads not clobber normal traffic.
- ajb 14y ago
  Probably because most consumer 'infrastructure' devices have razor-thin margins. This has two consequences: the vendor doesn't spend any time tweaking it, and second, the CPU is too puny to run all the packets through; they go instead through a hardware packet accelerator. It may be able to do the same thing, but not by configuring 'tc'. This is also why openwrt only runs sensibly (if at all) on the more expensive boxes.
alexchamberlain 14y ago
Could this be useful on web servers? If so, how?
nohup 14y ago
We created a complete bandwidth management product using iptables and tc back in 2001, with packages based on bandwidth, hours of usage, usage in hours etc. It was used by small ISPs. lartc.org was the bible we used to traverse the path of achieving control and command the precious resource called bandwidth. Those days are long gone, but building that small product gave me a chance to learn about networking like never before and taught me a lot. I owe a lot to lartc.org

224 points by powertower 14y ago

The little I've been able to find on the kernel and TC (Traffic Control) is here http://www.tldp.org/HOWTO/html_single/Traffic-Control-HOWTO/ http://www.tldp.org/HOWTO/html_single/Traffic-Control-HOWTO/

joshbaptiste 14y ago
indeed, The main reason tc(1) is underused is because it is under explained.
- nknight 14y ago
  Even were it adequately explained, it would still be underused, being a pretty crappy interface. I find it virtually the antithesis of iptables, whose syntax and meaning is fairly intuitive and easy both to remember, and derive from a few example rules.
  - JoachimSchipper 14y ago
    I agree that tc is not a very good user interface, but I'm not sure I agree with you that IPTables is. Consider http://www.openbsd.org/faq/pf/example1.html#allrules http://www.openbsd.org/faq/pf/example1.html#allrules - a nice configuration language does look better than a shell script.
    - zobzu 14y ago
      Well I happen to disagree with that. So yeah, the script is very small. But why is it? Because aliases are used everywhere. Over simplifying things. When you read it, you don't know what it does. When you read the 3 pages of explanations, you get the idea, but you still don't know exactly what it does. At least with iptables you do. And you can match virtually anything (like a bit in a packet if you like) The real interface issue is TC.
    - nknight 14y ago
      Uh, yeah, no, that's not my idea of a good user interface. If anything, you've just convinced me never to touch pf.
      - patrickgzill 14y ago
        Go ahead and touch it anyways. I can't imagine firewalling without it.

jen_h 14y ago

Have abused tc in the past to simulate high-latency remote dial-in connections while testing network equipment (netem delay ftw), but didn't realize that you could restrict by IP, thereby still being able to use the system for something other than dog-slow response. This is very cool - thanks for posting!

res0nat0r 14y ago

There really is a lack of good tc documentation out there and honestly interfacing with tc sucks. HSFC Scheduling with Linux http://linux-ip.net/articles/hfsc.en/ http://linux-ip.net/articles/hfsc.en/ is a good read.

fendale 14y ago

Quite a few years ago, I setup tc to do a bit of traffic shaping on my home network. Instead of having a standard router, I have a Linux box with two network cards using iptables to do NAT. I connect my cable modem to one of the network cards and then I connect my home network to the other. Way back, I found I had two problems - if I was doing something that did a lot of uploading (eg seeding a torrent), my download speed would tank because the tcp/ip packets were bogged down in the upload queue. I also like to be able to ssh into my machine while away from home, and if I did that while uploads were going on, the ssh terminal was basically unusable. Back then I discovered tc - it was able to make TCP control packets (ack's etc) and ssh related packets jump to the head of the queue, and the other upload stuff have a lower priority. Looking back at the script I created now, I don't really understand what it is doing, but it remember it worked. I still run it today and just take it for granted. I tested it by start a large upload while ssh'ed into the box from the internet - I turned off the tc rules and the terminal became unusably laggy, and then I turned the rules on and it worked perfectly.

sneak 14y ago
This helps a lot with that, as well: http://mosh.mit.edu/ http://mosh.mit.edu/
- Ralith 14y ago
  How would that help at all?
  - asharp 14y ago
    From my understanding, I think they meant that mosh would deal with the more congested conditions better then ssh, thus providing a better experience. Just a hunch though. That being said, mosh is rather cool ^_^
js2 14y ago
Once upon a time, I used to use http://lartc.org/wondershaper/ http://lartc.org/wondershaper/ for that. I never found it worked very well for me though.

sunyc 14y ago

it is under used because most of the time, you just don't have to care. and when it matters it lack the depth you need, like layer 7 detection. it is almost always better to just pay for bigger pipes.

rbanffy 14y ago
Wouldn't layer 7 detection drive CPU usage up for a somewhat different use case? You are not trying to give video traffic a lower priority - you are trying to prevent things like backups clogging your pipes.

lobster_johnson 14y ago

We use this with our office SDSL connection to prioritize normal traffic and deprioritize things like backups (which used to steal all the bandwidth and drive pings up to 1+ seconds). Works swell, highly recommended.

jcurbo 14y ago

tc is very useful for managing traffic over low bandwidth connections. I have used it to control traffic over satellite links (T1 scale throughput) with great success.

RobAtticus 14y ago

I've been using tc a lot recently in the research I've been doing and I have to agree that finding good documentation/examples is somewhat difficult to do. There are a lot of weird gotchas (the naming between uplink and downlink being one of them -- root vs ingress). Of course, I'm working on a better framework for controlling network resources, so I have to learn how tc works to provide a simpler interface for doing the things it does.

jvehent 14y ago
Simpler is better only if it provides the same level of flexibility. the TC command line is hard to learn at first, because it is extremely flexible... and after all, that's what you would expect from a packet scheduler.

KonradKlause 14y ago

Why underused? It's it well known and heavily used.

MatthewIfe 14y ago

As the author of the original answer, I have to agree that the interface is shambolic compared to iptables. I think much of this is due to using the same interface ideas as what is in the "ip" command which follows the same principles and is also underused in favour of ifconfig. They are actually part of the same package in fact. ifconfig has technically been deprecated in favour of ip for about 8 years. In any case, resource control on Linux is very very good these days. From traffic control to control groups (you can also manage tc classes in control groups too), its worth any system administrator having things like tc and cgroups in their arsenal so that the operating system controls the applications / users and not the other way around. A pain in the ass compared to other CLI interfaces I agree, but I personally place resource management high on my things to know about, in the same way I put access management and storage management there too.

secure 14y ago
I use 'ip' for all my network configuration/debugging and I think it’s a great tool. I can’t say the same for tc, which I tried to get into multiple times and failed. I will try it once more with your good explanation and understandable example. Thanks!

jvehent 14y ago

I wrote this: Journey to the Center of the Linux Kernel: Traffic Control, Shaping and QoS http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:networking:traffic_control http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:n...

herf 14y ago

I've used tc on my openwrt (Linksys!) firewall for years. A network genius friend helped set it up. Why are there no consumer firewalls with great defaults for this? How fantastic in the era of cloud storage and network backup to have long-running uploads not clobber normal traffic.

ajb 14y ago
Probably because most consumer 'infrastructure' devices have razor-thin margins. This has two consequences: the vendor doesn't spend any time tweaking it, and second, the CPU is too puny to run all the packets through; they go instead through a hardware packet accelerator. It may be able to do the same thing, but not by configuring 'tc'. This is also why openwrt only runs sensibly (if at all) on the more expensive boxes.

alexchamberlain 14y ago

Could this be useful on web servers? If so, how?

nohup 14y ago

We created a complete bandwidth management product using iptables and tc back in 2001, with packages based on bandwidth, hours of usage, usage in hours etc. It was used by small ISPs. lartc.org was the bible we used to traverse the path of achieving control and command the precious resource called bandwidth. Those days are long gone, but building that small product gave me a chance to learn about networking like never before and taught me a lot. I owe a lot to lartc.org