5 ms·

Many moons ago, support for TCP wrappers was removed from systemd -- and, a bit later, OpenSSH, IIRC. Fine, I understand. Personally, I still like tcpd (it's an

Many moons ago, support for TCP wrappers was removed from systemd -- and, a bit later, OpenSSH, IIRC. Fine, I understand. Personally, I still like tcpd (it's an extra layer) and I still use it... but I realize that not very many others do. Okay, I can live with this decision. Then, about two weeks ago, Red Hat decides they're gonna deprecate TCP wrappers, with a Red Hat employee saying: > TCP wrappers is a simple tool to block incoming connection on application level. This was very useful 20 years ago, when there were no firewalls in Linux. This is not the case for today and connection filtering should be done in network level or completely in application scope if it makes sense. Okay, fine. I still think it's useful as an extra layer but, nobody really uses it so, sure, let's get rid of it. I'm on board. Now, all of a sudden, this functionality shows up in systemd (announced yesterday)? Really!? This is (one of the) reason(s) why some of us greybeards loathe systemd (and just when I was starting to become "okay" with it, too).

icebraining 9y ago
Well, as the article points out, the systemd functionality is only a wrapper to the kernel's network filtering system (eBPF). It's not like they reimplemented TCPwrappers. You can also configure it without systemd. Julia Evans has a pretty good introduction to it: https://jvns.ca/blog/2017/04/07/xdp-bpf-tutorial/ https://jvns.ca/blog/2017/04/07/xdp-bpf-tutorial/
Jasper_ 9y ago
Because, as the engineer says, this filtering should be done at the network level, not with a TCP wrapper, a leaky userspace abstraction. systemd does this at the network level, with kernel APIs.
- marios 9y ago
  On the one hand, the user facing aspect of this feature as presented in this article seems really nice. On the other hand, it's another abstraction over rather new kernel APIs. Your average user may know of NetFilter kernel component, and the iptables userland counterpart. Some may even have heard of nftables. How many users know about eBPF/XDP ? More importantly, how many sysadmins know how to debug this when something goes wrong ? I'm aware there's a lot of development around eBPF and a number of players (Facebook, CloudFlare, ...) are already using XDP because of the ridiculous performance gain it provides for their use-cases. There's a myriad of frontends to iptables and yet they all suck. Each of them fail in mysterious ways and at that point the only way to debug what's wrong is to look at the ruleset with ... iptables. A package could ship with IP filtering in its service file, which may not match what the user expected. The symptom "service does not respond to TCP connections" is well known. In that case, it's common to check the process is alive (using ps), it has setup listening sockets (using netstat) and that the system firewall is not blocking it (using iptables/nftables or some frontend). Figuring out it's a eBPF/XDP misconfiguration is going to be painful IMHO. systemd attempts to cooperate with the system firewall. The subtlety is that there are now two facilities in the kernel that provide firewalling, and both can be on at the same time. To be honest, I don't even know which one has precedence. Why exactly does systemd need to mess with the networking services ? AFAIK, its primary function is that of a service manager; i.e: reliably bring services up and down. I feel all the rest should not be systemd's concern. After using systemd for quite some time, I've come to the conclusion that it works rather well when it sticks to starting/stopping services, assuming your service fits the systemd model. If it doesn't, debugging is extremely difficult and you often have to resort to dirty hacks to end up with a functionning system -- which is ironic considering one of systemd's motivations was to get rid of the dirty hacks in initscripts. An example of a buggy use case is when you need to pass environment variables to your executable and there's escaping involved. The systemd documentation suggests you can pass pretty much anything in the ExecStart lines. However, the reality is that it does not behave as other environments. The solution is to store the variables in a separate file, and not involve systemd at all. My takeaway is: why does systemd involve so many features ? It increases failure cases as well as the cognitive load required when dealing with it.
  - deleted 9y ago
    [deleted]
  - poettering 9y ago
    > Why exactly does systemd need to mess with the networking services? It doesn't. All this stuff is opt-in. If you don't want to use it, don't. What's a bit unfair though is that you imply that the various technologies where equivalent in behaviour and would be different only in performance. That's really not the case. The eBPF stuff allows configuration of per-cgroup, i.e. per-service or per-container access control. And that's a massive advantage. Classic Netfilter doesn't allow that: you can't tell it to permit all traffic to bittorrentd.service, because service/container/cgroup information is not available to it. Yes, people have tried to make that work, and there was some support for it in the kernel, but only for egress, and that made it pretty useless ultimately. You know, if iptables would allow us to configure tables per-service, we'd happily hook that up. Such a design is in fact where we started from originally. But the fact is that such a scheme isn't really compatible with Netfilter's design and thus didn't happen. And that that's the case is really not systemd's fault, but a technical decision of the kernel community. I know it is fashionable to blame systemd for everything, but seriously, eBPF+cgroups and all that kind of stuff is the result of the kernel's development scheme, and systemd only follows that and exposes what they came up with. If you don't like where kernel development goes, please complain about and to the kernel community, but systemd is just a consumer of what they do, and I couldn't care less if per-service firewalling is ultimately implemented based on netfilter, on nftables or on ebpf/cgroup. As it turns out only the latter ended up being capable enough to make this possible. And I accept that, and then made use of this. Do note that the port forward hookups in nspawn (i.e. --port=) are actually implemented using netfilter, not cgroups/ebpf. I hope that illustrates a bit that we use what works and is provided by the kernel community, and don't really follow an evil scheme here. Lennart
    - binaryapparatus 9y ago
      > It doesn't. All this stuff is opt-in. If you don't want to use it, don't. This sentence is getting rather long in a tooth. Cgroups? Kdbus? udev? Same mantra that is used over and over again turned out to be at least deceptive if not outright lie, each time it was used.
- hiciu 9y ago
  Just to expand this answers; this is not a tcp wrapper replacement. I mean, it can be used as one; but that's just a side effect. Since linux 4.10 you can attach eBPF programs to the cgroups: https://kernelnewbies.org/Linux_4.10#head-4f00d500db4f8e437a5ad6005d9cf39ee0124bec https://kernelnewbies.org/Linux_4.10#head-4f00d500db4f8e437a.... This allows you to do a really cool things, mostly firewalls attached to the cgroup and not to the system-wide netfilter stack. Imagine application-specific firewalls. Systemd uses that to add "simple" packet counting eBPF application to the cgroups: https://github.com/systemd/systemd/commit/1988a9d12015990c145a6e8515d5e22ef88b32cb https://github.com/systemd/systemd/commit/1988a9d12015990c14.... (eBPF is the same mechanism that is used with seccomp and filtering which syscalls can application use and which are forbidden - used in for example chromium sandbox on linux or with openssh)
binaryapparatus 9y ago
I don't know if I am more amazed with systemd engineering incompetence or the wider community that doesn't seem to see what's going on. I went from wondering why/what is going on to switching all my home/work/server machines to FreeBSD. Common response from the systemd groupies is to focus on technical bits and be precise but you can't focus on them if the concept/ideas/execution smell. Also, number of wrongly planned or wrongly executed bits counts in hundreds by now. Like cooking a frog.
- prrls 9y ago
  > I went from wondering why/what is going on to switching all my home/work/server machines to FreeBSD. Can you develop on some of the reasons that convinced you to run away from systemd ? I'm really curious to know why (honestly). Thanks :).
  - binaryapparatus 9y ago
    This is not even the best source of information but it shows the same trend I was spotting: https://suckless.org/sucks/systemd https://suckless.org/sucks/systemd There are similar sources on the internet that claim the same thing in different forms. Every single bad systemd decision is not terribly bad on its own, but when observed as a process they result in a horribly wrong and very insecure system. With systemd, there i crawling revolution of bad ideas, bad implementation and taking over of everything that was great in linux. When you try discussing any of those items separately you'll be redirected to 'technical discussion' and you can't really technically discuss conceptually wrong system. With bad concept it doesn't really matter how technically good or bad implementation is. Maybe the best telling thread, in order to understand how pushy and deceptive whole systemd is, is watching how Greg Kroah-Hartman tried to sneak kdbus into a kernel: https://lkml.org/lkml/2015/4/13/645 https://lkml.org/lkml/2015/4/13/645 It is very long mailing list thread and GH is not officially part of the systemd team -- only that he is, unofficially. He claims that kdbus brings great benefits and speedup only that he can't prove it. Every questioning of a very wrong concept is met by requesting of 'technical discussion'. If you have patience just read that lkml thread and you'll see the same sneaky politics of pushing things the same way that systemd people do it all the time.
poettering 9y ago
> Really!? This is (one of the) reason(s) why some of us greybeards loathe systemd (and just when I was starting to become "okay" with it, too). I am not sure why you imply that there was some evil scheme behind all this. In reality, things are a bit different: one group at Red Hat decided to put an end to TCP wrappers (which btw is something other distros already did a while back, we are late at the party), and another group then offered to make a more modern replacement available, that could cover some of the uses, in order not leave users in the cold. I am not sure why you think that's an evil thing to do? Isn't it nicer to drop something with a suggestion how to replace it then to drop it without replacement? Also, let's not forget: this functionality is implemented in the kernel, systemd merely adds knobs to expose this kernel functionality to the user. Hence, no, this functionality didn't really "show up in systemd". It showed up in the kernel, and systemd just wraps it to make it usable. I mean, this is a dilemma, isn't it? Whatever we do you are unhappy, right? ;-) Lennart
- reacweb 9y ago
  I do not think he is unhappy about this feature in systemd. He complains about the process and communication: removing a feature arguing it is useless then providing a replacement. Personally, I am very pleased with systemd and thanks to its novelty, it puts me on a same level as all the "greybeards". Continue the good work and please, take special care of the man pages.
- thwarted 9y ago
  I am not sure why you imply that there was some evil scheme behind all this. He didn't imply or say there was some evil scheme, and this reads like an attempt to play the victim card on your part. If no one uses the functionality is the reason for getting rid of it, then why add it back in? I seems like a lot of useless busy work to explicitly claim that no one uses something only to replace it.
- throwme211345 9y ago
  Lennart: you are a well paid tool and your designs and causes are not the causes of those of the prior generation of *nix users. When you understand that you will stop winking at us and understand that people will always be unhappy with your aggregations and the power that abets that aggregation. In the meantime we who can will look elsewhere and adopt and improve the model we understand and love because it is not broken and not improved by another design.
lscotte 9y ago
It's well worth educating yourself about eBPF, how it's functionality provided in the kernel (not systemd), but has only fully landed in the kernel recently. Hence, this timing makes perfect sense once you understand it.
thwarted 9y ago
Many moons ago, support for TCP wrappers was removed from systemd -- and, a bit later, OpenSSH, IIRC. connection filtering should be done in network level or completely in application scope if it makes sense. So it was done at the application scope, with a common config file format, and then it was deprecated because it should be done at the application scope? I can barely make sense of this logic they are applying. The only place it wasn't done at the application scope (baring an invocation wrapper that handled it for applications that didn't directly support tcpwrappers on its own listeners) was in systemd, because systemd isn't the end application, so if the logic is that it should be in the application scope, systemd is the only place it should have been removed from.

icebraining 9y ago

Well, as the article points out, the systemd functionality is only a wrapper to the kernel's network filtering system (eBPF). It's not like they reimplemented TCPwrappers. You can also configure it without systemd. Julia Evans has a pretty good introduction to it: https://jvns.ca/blog/2017/04/07/xdp-bpf-tutorial/ https://jvns.ca/blog/2017/04/07/xdp-bpf-tutorial/

Jasper_ 9y ago

Because, as the engineer says, this filtering should be done at the network level, not with a TCP wrapper, a leaky userspace abstraction. systemd does this at the network level, with kernel APIs.

marios 9y ago
On the one hand, the user facing aspect of this feature as presented in this article seems really nice. On the other hand, it's another abstraction over rather new kernel APIs. Your average user may know of NetFilter kernel component, and the iptables userland counterpart. Some may even have heard of nftables. How many users know about eBPF/XDP ? More importantly, how many sysadmins know how to debug this when something goes wrong ? I'm aware there's a lot of development around eBPF and a number of players (Facebook, CloudFlare, ...) are already using XDP because of the ridiculous performance gain it provides for their use-cases. There's a myriad of frontends to iptables and yet they all suck. Each of them fail in mysterious ways and at that point the only way to debug what's wrong is to look at the ruleset with ... iptables. A package could ship with IP filtering in its service file, which may not match what the user expected. The symptom "service does not respond to TCP connections" is well known. In that case, it's common to check the process is alive (using ps), it has setup listening sockets (using netstat) and that the system firewall is not blocking it (using iptables/nftables or some frontend). Figuring out it's a eBPF/XDP misconfiguration is going to be painful IMHO. systemd attempts to cooperate with the system firewall. The subtlety is that there are now two facilities in the kernel that provide firewalling, and both can be on at the same time. To be honest, I don't even know which one has precedence. Why exactly does systemd need to mess with the networking services ? AFAIK, its primary function is that of a service manager; i.e: reliably bring services up and down. I feel all the rest should not be systemd's concern. After using systemd for quite some time, I've come to the conclusion that it works rather well when it sticks to starting/stopping services, assuming your service fits the systemd model. If it doesn't, debugging is extremely difficult and you often have to resort to dirty hacks to end up with a functionning system -- which is ironic considering one of systemd's motivations was to get rid of the dirty hacks in initscripts. An example of a buggy use case is when you need to pass environment variables to your executable and there's escaping involved. The systemd documentation suggests you can pass pretty much anything in the ExecStart lines. However, the reality is that it does not behave as other environments. The solution is to store the variables in a separate file, and not involve systemd at all. My takeaway is: why does systemd involve so many features ? It increases failure cases as well as the cognitive load required when dealing with it.
- deleted 9y ago
  [deleted]
- poettering 9y ago
  > Why exactly does systemd need to mess with the networking services? It doesn't. All this stuff is opt-in. If you don't want to use it, don't. What's a bit unfair though is that you imply that the various technologies where equivalent in behaviour and would be different only in performance. That's really not the case. The eBPF stuff allows configuration of per-cgroup, i.e. per-service or per-container access control. And that's a massive advantage. Classic Netfilter doesn't allow that: you can't tell it to permit all traffic to bittorrentd.service, because service/container/cgroup information is not available to it. Yes, people have tried to make that work, and there was some support for it in the kernel, but only for egress, and that made it pretty useless ultimately. You know, if iptables would allow us to configure tables per-service, we'd happily hook that up. Such a design is in fact where we started from originally. But the fact is that such a scheme isn't really compatible with Netfilter's design and thus didn't happen. And that that's the case is really not systemd's fault, but a technical decision of the kernel community. I know it is fashionable to blame systemd for everything, but seriously, eBPF+cgroups and all that kind of stuff is the result of the kernel's development scheme, and systemd only follows that and exposes what they came up with. If you don't like where kernel development goes, please complain about and to the kernel community, but systemd is just a consumer of what they do, and I couldn't care less if per-service firewalling is ultimately implemented based on netfilter, on nftables or on ebpf/cgroup. As it turns out only the latter ended up being capable enough to make this possible. And I accept that, and then made use of this. Do note that the port forward hookups in nspawn (i.e. --port=) are actually implemented using netfilter, not cgroups/ebpf. I hope that illustrates a bit that we use what works and is provided by the kernel community, and don't really follow an evil scheme here. Lennart
  - binaryapparatus 9y ago
    > It doesn't. All this stuff is opt-in. If you don't want to use it, don't. This sentence is getting rather long in a tooth. Cgroups? Kdbus? udev? Same mantra that is used over and over again turned out to be at least deceptive if not outright lie, each time it was used.
hiciu 9y ago
Just to expand this answers; this is not a tcp wrapper replacement. I mean, it can be used as one; but that's just a side effect. Since linux 4.10 you can attach eBPF programs to the cgroups: https://kernelnewbies.org/Linux_4.10#head-4f00d500db4f8e437a5ad6005d9cf39ee0124bec https://kernelnewbies.org/Linux_4.10#head-4f00d500db4f8e437a.... This allows you to do a really cool things, mostly firewalls attached to the cgroup and not to the system-wide netfilter stack. Imagine application-specific firewalls. Systemd uses that to add "simple" packet counting eBPF application to the cgroups: https://github.com/systemd/systemd/commit/1988a9d12015990c145a6e8515d5e22ef88b32cb https://github.com/systemd/systemd/commit/1988a9d12015990c14.... (eBPF is the same mechanism that is used with seccomp and filtering which syscalls can application use and which are forbidden - used in for example chromium sandbox on linux or with openssh)

binaryapparatus 9y ago

I don't know if I am more amazed with systemd engineering incompetence or the wider community that doesn't seem to see what's going on. I went from wondering why/what is going on to switching all my home/work/server machines to FreeBSD. Common response from the systemd groupies is to focus on technical bits and be precise but you can't focus on them if the concept/ideas/execution smell. Also, number of wrongly planned or wrongly executed bits counts in hundreds by now. Like cooking a frog.

prrls 9y ago
> I went from wondering why/what is going on to switching all my home/work/server machines to FreeBSD. Can you develop on some of the reasons that convinced you to run away from systemd ? I'm really curious to know why (honestly). Thanks :).
- binaryapparatus 9y ago
  This is not even the best source of information but it shows the same trend I was spotting: https://suckless.org/sucks/systemd https://suckless.org/sucks/systemd There are similar sources on the internet that claim the same thing in different forms. Every single bad systemd decision is not terribly bad on its own, but when observed as a process they result in a horribly wrong and very insecure system. With systemd, there i crawling revolution of bad ideas, bad implementation and taking over of everything that was great in linux. When you try discussing any of those items separately you'll be redirected to 'technical discussion' and you can't really technically discuss conceptually wrong system. With bad concept it doesn't really matter how technically good or bad implementation is. Maybe the best telling thread, in order to understand how pushy and deceptive whole systemd is, is watching how Greg Kroah-Hartman tried to sneak kdbus into a kernel: https://lkml.org/lkml/2015/4/13/645 https://lkml.org/lkml/2015/4/13/645 It is very long mailing list thread and GH is not officially part of the systemd team -- only that he is, unofficially. He claims that kdbus brings great benefits and speedup only that he can't prove it. Every questioning of a very wrong concept is met by requesting of 'technical discussion'. If you have patience just read that lkml thread and you'll see the same sneaky politics of pushing things the same way that systemd people do it all the time.

poettering 9y ago

> Really!? This is (one of the) reason(s) why some of us greybeards loathe systemd (and just when I was starting to become "okay" with it, too). I am not sure why you imply that there was some evil scheme behind all this. In reality, things are a bit different: one group at Red Hat decided to put an end to TCP wrappers (which btw is something other distros already did a while back, we are late at the party), and another group then offered to make a more modern replacement available, that could cover some of the uses, in order not leave users in the cold. I am not sure why you think that's an evil thing to do? Isn't it nicer to drop something with a suggestion how to replace it then to drop it without replacement? Also, let's not forget: this functionality is implemented in the kernel, systemd merely adds knobs to expose this kernel functionality to the user. Hence, no, this functionality didn't really "show up in systemd". It showed up in the kernel, and systemd just wraps it to make it usable. I mean, this is a dilemma, isn't it? Whatever we do you are unhappy, right? ;-) Lennart

reacweb 9y ago
I do not think he is unhappy about this feature in systemd. He complains about the process and communication: removing a feature arguing it is useless then providing a replacement. Personally, I am very pleased with systemd and thanks to its novelty, it puts me on a same level as all the "greybeards". Continue the good work and please, take special care of the man pages.
thwarted 9y ago
I am not sure why you imply that there was some evil scheme behind all this. He didn't imply or say there was some evil scheme, and this reads like an attempt to play the victim card on your part. If no one uses the functionality is the reason for getting rid of it, then why add it back in? I seems like a lot of useless busy work to explicitly claim that no one uses something only to replace it.
throwme211345 9y ago
Lennart: you are a well paid tool and your designs and causes are not the causes of those of the prior generation of *nix users. When you understand that you will stop winking at us and understand that people will always be unhappy with your aggregations and the power that abets that aggregation. In the meantime we who can will look elsewhere and adopt and improve the model we understand and love because it is not broken and not improved by another design.

lscotte 9y ago

It's well worth educating yourself about eBPF, how it's functionality provided in the kernel (not systemd), but has only fully landed in the kernel recently. Hence, this timing makes perfect sense once you understand it.

thwarted 9y ago

Many moons ago, support for TCP wrappers was removed from systemd -- and, a bit later, OpenSSH, IIRC. connection filtering should be done in network level or completely in application scope if it makes sense. So it was done at the application scope, with a common config file format, and then it was deprecated because it should be done at the application scope? I can barely make sense of this logic they are applying. The only place it wasn't done at the application scope (baring an invocation wrapper that handled it for applications that didn't directly support tcpwrappers on its own listeners) was in systemd, because systemd isn't the end application, so if the logic is that it should be in the application scope, systemd is the only place it should have been removed from.