4 ms·

Is this a good time to define Poettering's Law - any simple and useful Unix service will eventually get rewritten into a monstrous uber service that requires a

Is this a good time to define Poettering's Law - any simple and useful Unix service will eventually get rewritten into a monstrous uber service that requires a supercomputer just to run, what should be, a simple service.

kzrdude 13y ago
You coin the law, and you can put your name on it. Let's see if it holds up.
FooBarWidget 13y ago
Which is easy for you to state, were it not for the fact that the "simple service" is broken in various ways. Let's start with the implementation-specific brokenness. I configured a log host with rsyslog. Rsyslog's configuration format, as well as documentation, is completely abysmal. And once in a while, rsyslog would use 100% CPU for no reason. A quick strace reveals that it keeps reading or writing to a file descriptor, but getting EAGAIN as error. Looks like a bug to me. And oh yeah, there's a race condition in rsyslog with regard to privilege lowering, so that sometimes it would great directories owned as root, and then it would lower its privilege, and then it would be unable to write to its own created directory. Just great. The above are all fixable. But what bothers me more is that syslog is entirely unauthenticated. Any process can write to syslog, claiming that it is any process. A malicious user can write a syslog entry saying that "init[1]: the system is unstable. please reboot now!". As an administrator, you will not be able to tell whether the message actually came from init (or PID 1) or not. A quick look at systemd's journal mechanism reveals systemd at least has thought about these problems: http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html http://www.freedesktop.org/software/systemd/man/systemd.jour.... It would log the process's PID, UID, GID and more, and the sender is unable to fake that.
- spc476 13y ago
  systemd can log the PID, UID and GID of a message because its using a Linux-specific feature of datagram based local sockets; such a feature cannot be supported on other Unix systems. But there are other reasons to use syslogd/rsyslogd/syslog-ng besides logging to local files, and that's to forward the logs to a centralized logging service. Of course, there are issues with that, as the default syslog protocol also allows any process, anywhere, to claim it is any other process (and if root, even fake its own source address) but it's being addressed (I know rsyslogd is, I'm not sure about other syslogd replacements). My issue with systemd is that forwarding logs to a central server is an afterthought, if even that. And the problem with that is that it won't be as secure or work as well if it's part of the design. And another issue with the "afterthought support for remote logging" is that I still need to run syslogd/rsyslogd/syslog-ng because I have a ton of other network equipment (routers) that can forward their logs via the syslog protocol (which I have used on networks I've managed---by doing that, I've been able to receive notifications of OSPF routing changes, for instance). And if I have syslogd/rsyslogd/syslog-ng already running, what are the benefits of systemd logging?
  - FooBarWidget 13y ago
    Regarding PID/UID/GID logging: if it has to use a Linux-specific feature to be less broken in that regard, then I'm all for it. In my opinion, authentication is such an important security feature that other Unices should just implement similar mechanisms (or, preferably, the same mechanism). Right now I've only seen the BSD people complaining that systemd uses Linux-specific features, but doing nothing to solve the problem. Regarding central logging: indeed, that is a problem. Has Lennart ever spoken about this issue?
    - spc476 13y ago
      The impression I've gotten is that he isn't interested in centralized logging, as that is outside of what he is trying to do. I could be wrong though.
- asveikau 13y ago
  Reading this I thought: how does one authenticate a pid or uid in a daemon? I am assuming Unix domain sockets, which don't provide you with the caller's pid or uid. Doing some googling I found there are indeed nonportable interfaces for this. First I found getpeerucred which is a Sun thing. Then I found Linux has a SO_PEERCRED socket option. [Edit: and there seems to exist getpeereid and LOCAL_CRED on some BSDs] Still seems like a weird or otherwise shaky thing to me. It feels like it breaks the abstraction of a socket.
  - spc476 13y ago
    Unix domain sockets are the way to pass open file descriptors between processes, though (why? I don't know, it just is). I think that because of that, the mechanism was extended to also pass along the PID, UID and GID. I did use the open file descriptor with SO_PEERCRED options to implement a Linux-only daemon to open privileged sockets on behalf of a process: https://github.com/spc476/ipacld https://github.com/spc476/ipacld (something I'm surprised has never been written before).
    - asveikau 13y ago
      I think the easier way is to create the socket with high privileges, then drop down to lower privilege.
      - spc476 13y ago
        But why have the keys to the entire kingdom when all you need is a single key to a single room? The daemon I wrote is very small, so there isn't much code to audit. The client that calls the daemon doesn't need to be root at all. The daemon can also make a lot of different checks (right UID, right GID, right executable image, etc.) before passing the privileged port back to the client. Edit: fixing the last sentence.
        FooBarWidget 13y ago
        A lot of stuff is built on the "worse is better" philosophy. Introducing a daemon like yours will complicate the OS a little bit more. So instead, OS developers outsource the complexity to the apps, and only implement a quick hack to make this possible. In practice, the running-as-root-then-drop-privilege pattern works well enough so nobody bothered making a more complex, but potentially more secure, solution.
mordae 13y ago
No, what happened was init scripts. Something completely alien to the original init tab concept. It was an extremely ugly hack that worked only because of extensive documentation and effort by much larger amount of people than systemd has now. Both systemd and journal are an actual solution. Try journalctl on your system, it actually makes sense! The silly thing even has bash completion of filtering criteria! And they are both lightning fast.
- zanny 13y ago
  I do want to mention binary logs piss me off, but I just set up journald to output to syslog and everything is fine. I just ignore journalctl. I personally find it to be quite slow, but I can just set it to not log its binary blobs.
- fosap 13y ago
  something that was not necessary. Init scripts can be nice and clean [1]. And yes, I'm all for a init replacement. Something new, nice and clean. But I take the shell script hackery over a dbus dependency. But that's just my personal opinion. I have neither used it nor looked at the code, but [2] seems to be what I want. It's by the libdietc author, so i expect the code to be small, elegant and not maintained very well. [1] http://www.freebsd.org/doc/en/articles/rc-scripting/rcng-dummy.html http://www.freebsd.org/doc/en/articles/rc-scripting/rcng-dum... [2] http://www.fefe.de/minit/ http://www.fefe.de/minit/
  - benbataille 13y ago
    That's again a myth. There is no dbus dependency. You can use dbus to do activation but you don't have to. Now, there is a dependency on libdbus which just means systemd is using the dbus serialisation format for its ipc (which, I repeat, don't have to happen over dbus).
    - fosap 13y ago
      Systemd seems to be quite mysterious, because the myth of the dbus dependency comes from the lead developer trying to debunk myths. http://0pointer.de/blog/projects/the-biggest-myths.html http://0pointer.de/blog/projects/the-biggest-myths.html >When you build systemd, it only requires three dependencies: glibc, libcap and dbus. In the paragraph "bloated".
zanny 13y ago
I use systemd and find it to be significantly faster boot-wise than Upstart and the sysv scripts. One doesn't do socket initialization and the other is a bunch of slow ass shell scripts. It doesn't matter if systemd is beefy if the actual task of starting services is lightning fast. Just for comparison, last year I was triple booting Debian Testing, Ubuntu 12.04, and Arch. I use KDE on all 3 (I do active development on KDE in the Arch one) and logged my boot times over 6 months. I misplaced those records but the effective boot times were around 15 seconds for Debian, 25 for Ubuntu, and 5 for Arch. Mainly because tasks in Ubuntu like NetworkManager or tmpfs mounting were taking 5 - 10 seconds and nothing else was running.

kzrdude 13y ago

You coin the law, and you can put your name on it. Let's see if it holds up.

FooBarWidget 13y ago

Which is easy for you to state, were it not for the fact that the "simple service" is broken in various ways. Let's start with the implementation-specific brokenness. I configured a log host with rsyslog. Rsyslog's configuration format, as well as documentation, is completely abysmal. And once in a while, rsyslog would use 100% CPU for no reason. A quick strace reveals that it keeps reading or writing to a file descriptor, but getting EAGAIN as error. Looks like a bug to me. And oh yeah, there's a race condition in rsyslog with regard to privilege lowering, so that sometimes it would great directories owned as root, and then it would lower its privilege, and then it would be unable to write to its own created directory. Just great. The above are all fixable. But what bothers me more is that syslog is entirely unauthenticated. Any process can write to syslog, claiming that it is any process. A malicious user can write a syslog entry saying that "init[1]: the system is unstable. please reboot now!". As an administrator, you will not be able to tell whether the message actually came from init (or PID 1) or not. A quick look at systemd's journal mechanism reveals systemd at least has thought about these problems: http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html http://www.freedesktop.org/software/systemd/man/systemd.jour.... It would log the process's PID, UID, GID and more, and the sender is unable to fake that.

spc476 13y ago
systemd can log the PID, UID and GID of a message because its using a Linux-specific feature of datagram based local sockets; such a feature cannot be supported on other Unix systems. But there are other reasons to use syslogd/rsyslogd/syslog-ng besides logging to local files, and that's to forward the logs to a centralized logging service. Of course, there are issues with that, as the default syslog protocol also allows any process, anywhere, to claim it is any other process (and if root, even fake its own source address) but it's being addressed (I know rsyslogd is, I'm not sure about other syslogd replacements). My issue with systemd is that forwarding logs to a central server is an afterthought, if even that. And the problem with that is that it won't be as secure or work as well if it's part of the design. And another issue with the "afterthought support for remote logging" is that I still need to run syslogd/rsyslogd/syslog-ng because I have a ton of other network equipment (routers) that can forward their logs via the syslog protocol (which I have used on networks I've managed---by doing that, I've been able to receive notifications of OSPF routing changes, for instance). And if I have syslogd/rsyslogd/syslog-ng already running, what are the benefits of systemd logging?
- FooBarWidget 13y ago
  Regarding PID/UID/GID logging: if it has to use a Linux-specific feature to be less broken in that regard, then I'm all for it. In my opinion, authentication is such an important security feature that other Unices should just implement similar mechanisms (or, preferably, the same mechanism). Right now I've only seen the BSD people complaining that systemd uses Linux-specific features, but doing nothing to solve the problem. Regarding central logging: indeed, that is a problem. Has Lennart ever spoken about this issue?
  - spc476 13y ago
    The impression I've gotten is that he isn't interested in centralized logging, as that is outside of what he is trying to do. I could be wrong though.
asveikau 13y ago
Reading this I thought: how does one authenticate a pid or uid in a daemon? I am assuming Unix domain sockets, which don't provide you with the caller's pid or uid. Doing some googling I found there are indeed nonportable interfaces for this. First I found getpeerucred which is a Sun thing. Then I found Linux has a SO_PEERCRED socket option. [Edit: and there seems to exist getpeereid and LOCAL_CRED on some BSDs] Still seems like a weird or otherwise shaky thing to me. It feels like it breaks the abstraction of a socket.
- spc476 13y ago
  Unix domain sockets are the way to pass open file descriptors between processes, though (why? I don't know, it just is). I think that because of that, the mechanism was extended to also pass along the PID, UID and GID. I did use the open file descriptor with SO_PEERCRED options to implement a Linux-only daemon to open privileged sockets on behalf of a process: https://github.com/spc476/ipacld https://github.com/spc476/ipacld (something I'm surprised has never been written before).
  - asveikau 13y ago
    I think the easier way is to create the socket with high privileges, then drop down to lower privilege.
    - spc476 13y ago
      But why have the keys to the entire kingdom when all you need is a single key to a single room? The daemon I wrote is very small, so there isn't much code to audit. The client that calls the daemon doesn't need to be root at all. The daemon can also make a lot of different checks (right UID, right GID, right executable image, etc.) before passing the privileged port back to the client. Edit: fixing the last sentence.
      - FooBarWidget 13y ago
        A lot of stuff is built on the "worse is better" philosophy. Introducing a daemon like yours will complicate the OS a little bit more. So instead, OS developers outsource the complexity to the apps, and only implement a quick hack to make this possible. In practice, the running-as-root-then-drop-privilege pattern works well enough so nobody bothered making a more complex, but potentially more secure, solution.

mordae 13y ago

No, what happened was init scripts. Something completely alien to the original init tab concept. It was an extremely ugly hack that worked only because of extensive documentation and effort by much larger amount of people than systemd has now. Both systemd and journal are an actual solution. Try journalctl on your system, it actually makes sense! The silly thing even has bash completion of filtering criteria! And they are both lightning fast.

zanny 13y ago
I do want to mention binary logs piss me off, but I just set up journald to output to syslog and everything is fine. I just ignore journalctl. I personally find it to be quite slow, but I can just set it to not log its binary blobs.
fosap 13y ago
something that was not necessary. Init scripts can be nice and clean [1]. And yes, I'm all for a init replacement. Something new, nice and clean. But I take the shell script hackery over a dbus dependency. But that's just my personal opinion. I have neither used it nor looked at the code, but [2] seems to be what I want. It's by the libdietc author, so i expect the code to be small, elegant and not maintained very well. [1] http://www.freebsd.org/doc/en/articles/rc-scripting/rcng-dummy.html http://www.freebsd.org/doc/en/articles/rc-scripting/rcng-dum... [2] http://www.fefe.de/minit/ http://www.fefe.de/minit/
- benbataille 13y ago
  That's again a myth. There is no dbus dependency. You can use dbus to do activation but you don't have to. Now, there is a dependency on libdbus which just means systemd is using the dbus serialisation format for its ipc (which, I repeat, don't have to happen over dbus).
  - fosap 13y ago
    Systemd seems to be quite mysterious, because the myth of the dbus dependency comes from the lead developer trying to debunk myths. http://0pointer.de/blog/projects/the-biggest-myths.html http://0pointer.de/blog/projects/the-biggest-myths.html >When you build systemd, it only requires three dependencies: glibc, libcap and dbus. In the paragraph "bloated".

zanny 13y ago

I use systemd and find it to be significantly faster boot-wise than Upstart and the sysv scripts. One doesn't do socket initialization and the other is a bunch of slow ass shell scripts. It doesn't matter if systemd is beefy if the actual task of starting services is lightning fast. Just for comparison, last year I was triple booting Debian Testing, Ubuntu 12.04, and Arch. I use KDE on all 3 (I do active development on KDE in the Arch one) and logged my boot times over 6 months. I misplaced those records but the effective boot times were around 15 seconds for Debian, 25 for Ubuntu, and 5 for Arch. Mainly because tasks in Ubuntu like NetworkManager or tmpfs mounting were taking 5 - 10 seconds and nothing else was running.