9 ms·

> replacing a cumbersome collection of shell scripts that previously were used for init/service management. systemd-as-init was fine. systemd replacing udevd,

> replacing a cumbersome collection of shell scripts that previously were used for init/service management. systemd-as-init was fine. systemd replacing udevd, syslog†, mount, ntpd, DNS, dhcpd, and probably a bunch more stuff I've forgotten or don't know about, is something else. If I wanted a tightly-coupled opaque binary blob of a system I'd run Windows. † Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no. ‡ And journald also cannot send logs off-host in any useful format, so I end up having to run rsyslogd anyway to get things into my SIEM through something useful like RFC 3164 or 5424. But yeah: unit files are (non-sarcastically) pretty good.

ahartmetz 6y ago
In addition to the gratuitous tight coupling, the quality of implementation of most of these system service replacements is pretty bad! NTP and DNS clients are clearly worse (more limited and more error-prone and buggy than the separate packages they replace). The syslog replacement is only not a major issue for me because my distribution (K)Ubuntu has it configured to produce plaintext log files in addition to the binary files. I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively. That said, the OOM killer daemon seems to be a good thing.
- vetinari 6y ago
  systemd-resolved and dnsmasq are the only two clients that can properly route DNS queries via the proper interface to their respective authoritative server based on the resolved hostname itself. It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq. The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.
  - ahartmetz 6y ago
    Well. This http://0pointer.de/blog/projects/socket-activation.html http://0pointer.de/blog/projects/socket-activation.html sells socket activation as the best thing since sliced bread, including for boot time. It has its uses for some rarely used services and for some startup order problems, yes. About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. Can't find it right now. I have disabled it on my system as well due to a problem I don't remember. Maybe it didn't use my router's DNS server and went straight to the internet. Maybe I just wanted it to leave my resolv.conf alone.
    - vetinari 6y ago
      > About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. It was a fallback DNS used when the user has configured nothing. Not only that, it was a default fallback that can be changed by both the distribution package and the user (including disabling completely). It was a tempest in the teapot, by people who have to find some drama.
      - ahartmetz 6y ago
        I think I remember what it was now. Local configuration (DHCP?) contained two DNS servers. systemd-resolved switched to the global fallback when it tried one of the local servers and it failed, instead of falling back to the other local server first. It's a completely reasonable configuration to have two local servers for redundancy as well as load balancing, and systemd-resolved broke the redundancy aspect. It was not about a "wrong" global fallback, but about using it when a local alternative still works. It breaks local network name resolution.
        6y ago
anotherhue 6y ago
To be fair, there isn't one Uber-bin, but that doesn't change the fact that all these systems are now in lock step and under the control of one group. Tell me, a naive one, why a throwaway for this?
- throw0101a 6y ago
  > To be fair, there isn't one Uber-bin Damning with faint praise? :) While an uber-bin would be a bigger problem, there is a reason why I italicized "tightly-coupled". Often many people simply respond with "but there are multiple binaries",† which kind of misses the point. > Tell me, a naive one, why a throwaway for this? It was created 2017-01-01. † Edit: Case in point, see sibling comment: "These are separate services, with separate binaries, with separate packages […]" * https://news.ycombinator.com/item?id=26646250 https://news.ycombinator.com/item?id=26646250
- tored 6y ago
  > under the control of one group Exactly.
- barkingcat 6y ago
  Why do you assume a throwaway?
vetinari 6y ago
>... ntpd, DNS, dhcpd These are separate services, with separate binaries, with separate packages for most distributions and their use is optional; they have their uses. For example, most distributions won't use systemd-networkd, because it is (intentionally) quite limited and they use NetworkManager anyway. RHEL8 for example doesn't even ship a package systemd-networkd.
- throw0101a 6y ago
  > These are separate services, with separate binaries, with separate packages Then why are they in the same repo as the udevd code? And why did udevd have to be pulled into the same repo as the init code when it was doing just fine outside of it?
  - vetinari 6y ago
    Probably because it was not doing just fine outside of it. If you think they took the wrong approach and yours is the right one, show them by doing.
    - throw0101a 6y ago
      > If you think they took the wrong approach and yours is the right one, show them by doing. If Roger Ebert said a movie was awful, did people expect him to write, produce, direct, and/or act in one? If Doug DeMuro said a car handles badly, are people expecting him to get a mechanical engineering degree and build a better one?
      - faho 6y ago
        Ebert did write a movie: https://en.m.wikipedia.org/wiki/Beyond_the_Valley_of_the_Dolls https://en.m.wikipedia.org/wiki/Beyond_the_Valley_of_the_Dol...
        throw0101a 6y ago
        Yes, I know. :) And that would probably be the level of quality of any code I tried to write. :)
ragnese 6y ago
I definitely understand that criticism w.r.t. all of the systemd-verse replacement tools. I'm not an expert. I'm just a software dev and free software nut who wants a free software OS. But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev? You say, disparagingly, that it is tightly-coupled and opaque (binary). Why is loosely coupled better in these cases specifically? Weren't `mount`, `ntpd`, `udev`, etc all compiled C code already? How does systmed make those service more opaque? Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
- throw0101a 6y ago
  > But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev? What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it. But it stood on its own merits, was able to mature over time, and when people thought it good, they could willing start using it (or not). > Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
  - tkfu 6y ago
    > What does a time system have to with system start infrastructure? Accurate time is vital for many cryptographic operations. For example, in the embedded space you may want to have a check that your current disk image (ideally checked by dm-verity) is actually up to date. And to verify the metadata signing that disk image using Uptane/TUF/SUIT or similar protocols you need verifiable time. Or you might need to contact a provisioning server at boot time, and need to be able to verify that server's TLS cert. (Now, systemd used to have a bug[1] where time-sync.target would be reached before time was actually synced, and Poettering's response to the bug was the all-too-typical typical "yeah, don't worry about it, it's intended behaviour that the time-sync target doesn't mean that time is synced"[2]. But that did get fixed a couple years ago [3].) [1] https://github.com/systemd/systemd/issues/5097 https://github.com/systemd/systemd/issues/5097 [2] https://github.com/systemd/systemd/issues/5097#issuecomment-276787961 https://github.com/systemd/systemd/issues/5097#issuecomment-... [3] https://github.com/systemd/systemd/pull/8494 https://github.com/systemd/systemd/pull/8494
pengaru 6y ago
> † Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no. When current journals are detected as dirty by journald @ startup, they are renamed and no longer written to. This is a very robust and conservative approach. It burns some space in the interest of simplicity and preserving the log data without risk of messing things up. These renamed journals are still accessed when reading, so it's not like those dirty files no longer participate in journalctl operations; their contents are not lost. Any logs lost in a crash are just what was in-flight IPC, or sitting dirty in the kernel's buffer cache and hadn't been sync'd to backing store yet. Journald also performs an explicit sync whenever an urgent message arrives, to try ensure it's made durable ASAP. Of course there's still a filesystem and potentially myriad layers below that which could lose data in a crash. ACID would be overkill for this fairly simple single-writer multiple-reader mostly-appended situation. Edit: Your statement "when it poops its pants and corrupts the file" implies journald has a tendency to actively corrupt its own files. The principal mechanism journald uses to detect potentially corrupt files is simply identifying a file as being ONLINE when opening for writing, typically at startup. This is almost always caused by an unclean shutdown of the host, though it could be due to a crash of journald itself. The file isn't even necessarily corrupt. It's simply assumed inconsistent and treated as such WRT writability. I'm not aware of any journald bugs in recent history where journald actively corrupted the contents of journal files. Even crashy bugs where journald was littering dirty journals to be treated as corrupt I'm not aware of being actually corrupted internally by journald bugs. Source: I've worked quite a bit on journald over the years
Blikkentrekker 6y ago
> But yeah: unit files are (non-sarcastically) pretty good. I personally like service managers more that have clean shell scripts such as runit or OpenRC where shell scripts more so look like this: #!/usr/bin/openrc-run depends="some list of services to depend on" after="some list of services which need to start earlier" cmd="/path/to/executable" args="command line arguments" The nice thing is that it's a shell script, so each of these lines can be defined conditionally.
- throw0101a 6y ago
  See also the BSDs, which have similarly simple scripts
nijave 6y ago
I'd be interested in how your logs actually got corrupted (was journald to blame or was there power/filesystems issues that were just hard to recover from) I actually like journald quite a bit. Before, you'd have log files scattered all over the place with various cronjobs to try to keep them from filling up the disk (usually with the help of logrotate). Each application was free to put its logs wherever it decided. Journald keeps them structured with first-class support for metadata like timestamp and unit so it's easy to get a service's logs in a 2 hour window on a given day and equally as easy to look at them in a stream with other service's logs (it aggregates them for you). From an application perspective, you just dump to stdout/err. >And journald also cannot send logs off-host in any useful format This seems directly at odds with your "tightly-coupled" beginning statement (you want it to do more or you think it does too much?). It's trivial to do this with fluentd or other log aggregation software

ahartmetz 6y ago

In addition to the gratuitous tight coupling, the quality of implementation of most of these system service replacements is pretty bad! NTP and DNS clients are clearly worse (more limited and more error-prone and buggy than the separate packages they replace). The syslog replacement is only not a major issue for me because my distribution (K)Ubuntu has it configured to produce plaintext log files in addition to the binary files. I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively. That said, the OOM killer daemon seems to be a good thing.

vetinari 6y ago
systemd-resolved and dnsmasq are the only two clients that can properly route DNS queries via the proper interface to their respective authoritative server based on the resolved hostname itself. It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq. The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.
- ahartmetz 6y ago
  Well. This http://0pointer.de/blog/projects/socket-activation.html http://0pointer.de/blog/projects/socket-activation.html sells socket activation as the best thing since sliced bread, including for boot time. It has its uses for some rarely used services and for some startup order problems, yes. About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. Can't find it right now. I have disabled it on my system as well due to a problem I don't remember. Maybe it didn't use my router's DNS server and went straight to the internet. Maybe I just wanted it to leave my resolv.conf alone.
  - vetinari 6y ago
    > About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. It was a fallback DNS used when the user has configured nothing. Not only that, it was a default fallback that can be changed by both the distribution package and the user (including disabling completely). It was a tempest in the teapot, by people who have to find some drama.
    - ahartmetz 6y ago
      I think I remember what it was now. Local configuration (DHCP?) contained two DNS servers. systemd-resolved switched to the global fallback when it tried one of the local servers and it failed, instead of falling back to the other local server first. It's a completely reasonable configuration to have two local servers for redundancy as well as load balancing, and systemd-resolved broke the redundancy aspect. It was not about a "wrong" global fallback, but about using it when a local alternative still works. It breaks local network name resolution.
      - 6y ago

anotherhue 6y ago

To be fair, there isn't one Uber-bin, but that doesn't change the fact that all these systems are now in lock step and under the control of one group. Tell me, a naive one, why a throwaway for this?

throw0101a 6y ago
> To be fair, there isn't one Uber-bin Damning with faint praise? :) While an uber-bin would be a bigger problem, there is a reason why I italicized "tightly-coupled". Often many people simply respond with "but there are multiple binaries",† which kind of misses the point. > Tell me, a naive one, why a throwaway for this? It was created 2017-01-01. † Edit: Case in point, see sibling comment: "These are separate services, with separate binaries, with separate packages […]" * https://news.ycombinator.com/item?id=26646250 https://news.ycombinator.com/item?id=26646250
tored 6y ago
> under the control of one group Exactly.
barkingcat 6y ago
Why do you assume a throwaway?

vetinari 6y ago

>... ntpd, DNS, dhcpd These are separate services, with separate binaries, with separate packages for most distributions and their use is optional; they have their uses. For example, most distributions won't use systemd-networkd, because it is (intentionally) quite limited and they use NetworkManager anyway. RHEL8 for example doesn't even ship a package systemd-networkd.

throw0101a 6y ago
> These are separate services, with separate binaries, with separate packages Then why are they in the same repo as the udevd code? And why did udevd have to be pulled into the same repo as the init code when it was doing just fine outside of it?
- vetinari 6y ago
  Probably because it was not doing just fine outside of it. If you think they took the wrong approach and yours is the right one, show them by doing.
  - throw0101a 6y ago
    > If you think they took the wrong approach and yours is the right one, show them by doing. If Roger Ebert said a movie was awful, did people expect him to write, produce, direct, and/or act in one? If Doug DeMuro said a car handles badly, are people expecting him to get a mechanical engineering degree and build a better one?
    - faho 6y ago
      Ebert did write a movie: https://en.m.wikipedia.org/wiki/Beyond_the_Valley_of_the_Dolls https://en.m.wikipedia.org/wiki/Beyond_the_Valley_of_the_Dol...
      - throw0101a 6y ago
        Yes, I know. :) And that would probably be the level of quality of any code I tried to write. :)

ragnese 6y ago

I definitely understand that criticism w.r.t. all of the systemd-verse replacement tools. I'm not an expert. I'm just a software dev and free software nut who wants a free software OS. But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev? You say, disparagingly, that it is tightly-coupled and opaque (binary). Why is loosely coupled better in these cases specifically? Weren't `mount`, `ntpd`, `udev`, etc all compiled C code already? How does systmed make those service more opaque? Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?

throw0101a 6y ago
> But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev? What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it. But it stood on its own merits, was able to mature over time, and when people thought it good, they could willing start using it (or not). > Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
- tkfu 6y ago
  > What does a time system have to with system start infrastructure? Accurate time is vital for many cryptographic operations. For example, in the embedded space you may want to have a check that your current disk image (ideally checked by dm-verity) is actually up to date. And to verify the metadata signing that disk image using Uptane/TUF/SUIT or similar protocols you need verifiable time. Or you might need to contact a provisioning server at boot time, and need to be able to verify that server's TLS cert. (Now, systemd used to have a bug[1] where time-sync.target would be reached before time was actually synced, and Poettering's response to the bug was the all-too-typical typical "yeah, don't worry about it, it's intended behaviour that the time-sync target doesn't mean that time is synced"[2]. But that did get fixed a couple years ago [3].) [1] https://github.com/systemd/systemd/issues/5097 https://github.com/systemd/systemd/issues/5097 [2] https://github.com/systemd/systemd/issues/5097#issuecomment-276787961 https://github.com/systemd/systemd/issues/5097#issuecomment-... [3] https://github.com/systemd/systemd/pull/8494 https://github.com/systemd/systemd/pull/8494

pengaru 6y ago

> † Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no. When current journals are detected as dirty by journald @ startup, they are renamed and no longer written to. This is a very robust and conservative approach. It burns some space in the interest of simplicity and preserving the log data without risk of messing things up. These renamed journals are still accessed when reading, so it's not like those dirty files no longer participate in journalctl operations; their contents are not lost. Any logs lost in a crash are just what was in-flight IPC, or sitting dirty in the kernel's buffer cache and hadn't been sync'd to backing store yet. Journald also performs an explicit sync whenever an urgent message arrives, to try ensure it's made durable ASAP. Of course there's still a filesystem and potentially myriad layers below that which could lose data in a crash. ACID would be overkill for this fairly simple single-writer multiple-reader mostly-appended situation. Edit: Your statement "when it poops its pants and corrupts the file" implies journald has a tendency to actively corrupt its own files. The principal mechanism journald uses to detect potentially corrupt files is simply identifying a file as being ONLINE when opening for writing, typically at startup. This is almost always caused by an unclean shutdown of the host, though it could be due to a crash of journald itself. The file isn't even necessarily corrupt. It's simply assumed inconsistent and treated as such WRT writability. I'm not aware of any journald bugs in recent history where journald actively corrupted the contents of journal files. Even crashy bugs where journald was littering dirty journals to be treated as corrupt I'm not aware of being actually corrupted internally by journald bugs. Source: I've worked quite a bit on journald over the years

Blikkentrekker 6y ago

> But yeah: unit files are (non-sarcastically) pretty good. I personally like service managers more that have clean shell scripts such as runit or OpenRC where shell scripts more so look like this: #!/usr/bin/openrc-run depends="some list of services to depend on" after="some list of services which need to start earlier" cmd="/path/to/executable" args="command line arguments" The nice thing is that it's a shell script, so each of these lines can be defined conditionally.

throw0101a 6y ago
See also the BSDs, which have similarly simple scripts

nijave 6y ago

I'd be interested in how your logs actually got corrupted (was journald to blame or was there power/filesystems issues that were just hard to recover from) I actually like journald quite a bit. Before, you'd have log files scattered all over the place with various cronjobs to try to keep them from filling up the disk (usually with the help of logrotate). Each application was free to put its logs wherever it decided. Journald keeps them structured with first-class support for metadata like timestamp and unit so it's easy to get a service's logs in a 2 hour window on a given day and equally as easy to look at them in a stream with other service's logs (it aggregates them for you). From an application perspective, you just dump to stdout/err. >And journald also cannot send logs off-host in any useful format This seems directly at odds with your "tightly-coupled" beginning statement (you want it to do more or you think it does too much?). It's trivial to do this with fluentd or other log aggregation software