26 ms·

I get the impression that OP seems to just be abandoning systemd to build another... systemd. So it seems like not-invented-here syndrome. I don't really know

by atonse 9y ago

I get the impression that OP seems to just be abandoning systemd to build another... systemd. So it seems like not-invented-here syndrome. I don't really know about any of the drama surrounding systemd but I've liked using it so far. It's easily the most attractive feature in Ubuntu 16.04 LTS. Using journalctl for logs is nice because I don't have to keep looking around for where logs are stored and then tail them. But then again, I don't worry about log rotation and things like that. My guess is that journalctl takes care of it. I'm sure I've barely scratched the surface of all the things it can do (like auto-restarting daemons, some security stuff, etc). Am I the exception? Can't be so, if most linux distros have adopted systemd.

ragebol 9y ago
> [systemd] easily the most attractive feature in Ubuntu 16.04 LTS. For me it is as well, I really miss it when working with a Ubuntu 14.04 box. I have not found a single downside to systemd, really.
usernam 9y ago
systemd is not perfect. Debugging systemd is still very hard. There is no way to simulate what actions would be performed to reach a target without actually executing them. There are plenty of issues related to boot and shutdown which are _hard_ to track down because of this. The journal is nice, especially on desktop systems, but it's way too slow to be used in any moderately busy system. The disk space utilization is also _prohibitive_ for archival. We still rely on remote syslog because of this, but it's not without issues: the journal->syslog translation is lossy, and it's compounded with some "quirky" upstream decisions (purely cosmetic) to write unit descriptions in the log as opposed to the real unit name which make tracking down units harder if you don't have the actual journal file. Journal files also get corrupted easily with unclean shutdown. In case of issues (from software to hardware), the local journal quickly becomes completely useless. All in all, systemd is an improvement, but it lacks the debugging tools, performance and overall polish you would expect from a central piece of software such as init. The internal APIs are mostly undocumented or self-referential. The DBus interface is opaque. Things gets muddy very fast as soon as you touch non-central components such as systemd-resolved or networkd. I'm not a detractor, but there are many things systemd can still improve, but it feels we're kind of stuck. I'm quite happy if we have some competition here.
- majewsky 9y ago
  > Journal files also get corrupted easily with unclean shutdown. Huh? I thought that they introduced this weird binary log format precisely to avoid this kind of error. (I don't remember the exact reasoning, though.)
  - usernam 9y ago
    You're confusing the binary format with the tamper protection, or file "sealing". The two features are really independent.
  - marcosdumay 9y ago
    "They" were not very successful in this, like they weren't in any other stability related feature.
- cmurf 9y ago
  The journal files can get corrupted, but it's a localized corruption that journalctl skips over and continues on without additional data loss. Obviously anything that didn't completely commit is gone, which would be gone anyway with some other format. The main difference is that systemd-journald can a.) become aware of the corruption, b.) warn of it rather than display corrupt info, and c.) recover and continue on. There have been a pile of bugs I've run into and reported where journald gets really confused; e.g. 'journalctl -b-5' won't actually go back five boots, it'll show some other content, or it'll give an error, but using --since will work fine. I haven't seen this problem recently, so the fix may be trickling out.
  - bryanlarsen 9y ago
    That's not my experience -- once corrupted it stays corrupted. Is this something that's been fixed since Ubuntu 16.04?
    - cmurf 9y ago
      I don't know what you're contradicting because I never said the corruption gets fixed. I said it's skipped over. Specifically what happens is upon detection of corruption, journald stops writing to the corrupt journal file, creates a new journal file, and new entries go there. The utility for reading the journal files, journalctl, can also detect this corruption and will skip over detected corruption and continue on. I have no idea what version of systemd Ubuntu 16.04 has, but in systemd-233 I'm not seeing aforementioned problems with -b switch and corrupt journal files.
gbromios 9y ago
>I get the impression that OP seems to just be abandoning systemd to build another... systemd. So it seems like not-invented-here syndrome. well, he does say: >There may be a bit of NIH syndrome leading to this decision; that’s ok, I can live with that; >I don't really know about any of the drama surrounding systemd I have only been a passive observer of said drama, it boils down to A) init was a stable, widely used, and well understood system that didn't need an update B) systemd takes on far too much responsibility, contrary to the unix philosophy. personally, I have experienced only minor annoyance when having to re-learn familiar commands, though there seems to be a init compatibility layer that has made it pretty transparent to me. It'd probably have been more annoying if I were still an admin.
- lmz 9y ago
  > init was a stable, widely used, and well understood system that didn't need an update If it didn't need an update why did various parties try and replace it e.g. OSX launchd, Solaris SMF, Ubuntu Upstart?
  - gbromios 9y ago
    I have no idea. Like I said, I was only a casual observer of the rally against systemd. I get the feeling that the opposition was more about ethos than any irrefutable technical argument.
    - briffle 9y ago
      Init also took a VERY long time to start, and had very, very rudimentary concepts of dependencies.
  - petre 9y ago
    Launchd is actually quite a hurdle to use. XML service files, inconsistent syntax for launchctl commands etc. Apple probably designed it so .plist files would be editable with some GUI tool.
stuaxo 9y ago
I was building a system on raspberry pi involving a few daemons + things that needed starting on powerup + 24h after powerup + before powerdown. It made all these things really easy, I'm not sure why everyone is so keen on doing everything with various bash scripts like we used to.
- JdeBP 9y ago
  You are making the oft-made error of thinking that System 5 rc was the precursor to systemd. On several major systems it was not. The precursor was upstart on Fedora and Ubuntu, for example. See http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ for the System 5 rc fallacy and several others to avoid.
vidarh 9y ago
I'm sure there's room for reinventing a better systemd, but most of the attempts I've seen either fail to understand and intentionally ignore just how many things systemd simplify. I can understand wanting to decouple and simplify that, but I think a lot of the time the people trying to reinvent it do not understand the concerns that systemd solves for people. E.g. a typical issue is process restarting. Simple. Can be done in a couple of dozen lines. Except that solves nothing for most people: If your server is spinning in place trying to start your big-proper-process-monitor, and it's failing to log anything because rsyslogd or whatever is also just spinning in place, and the boot processes that should have mounted the filesystems needed to get things working is repeatedly crashing? You can mitigate all of that, but now you're creating a separate mechanism (maybe as simple as writing to disk, but still) for early-stage logging, and a separate mechanism and your init needs to deal with things like throttling restart and handling them and so does a separate process manager, and so on. In order to simplify things, there's duplication of functionality all over the place. A huge part of the architecture of systemd is down to starting to pull at that thread and untangling all of these issues. Is it perfect? No. Is it the solution to all of the concerns they've uncovered? Probably not. But what we had before was so much worse, and I'm so happy someone have had a go at addressing them. And frankly, it's great that animosity towards systemd has motivated so many people to finally try to address this issues in other ways. Maybe one of them eventually solves what none of them did before systemd forced peoples hand.
- captainmuon 9y ago
  I think for many people who are dissatisfied or angry, systemd didn't simplify things. Many people have simple use cases, and for them it made things much more complicated. Monolithic, incomprehensible, unintrospectable. It is like a modern car engine which is just a sealed black box under the hood, full of electronics. Sure, it works great, it is progress compared to before, everything is there for a reason. But you can't easily fix things yourself. And for many people, the old kind of engine was good enough. I mean, for my simple uses, what is wrong with `sudo service nginx restart`, or `sudo killall nginx`? For many people, it is really dead simple. I get that it is not simple for everybody, but that is why we should have choice, not one-size-fits-all.
captainmuon 9y ago
That's fine to me. Linux is about choice. One of my biggest problems with systemd is that other stuff starts to depend on it. For things like GNOME, it is not just an init system, it is THE one init system. Also, I'm sure journald is useful, but I've had so many problems with it. For me as a Desktop and occasional remote computing, small server user, logs are ephemeral. Something goes wrong, I can quickly grep the logs which are plain text. I didn't really feel any advantage from the modern, binary, structured logs. Also, I spent hours installing something on WSL (Linux on Windows), finding that everything works except for the systemd logs, which are empty. The command that controls services spits out hundreds of cryptic lines, services interweaved with all kinds of other objects, when I just want to list the running services. It has too many moving parts for my taste. I don't need socket activation. I don't need a separate DNS resolver in my init system (unrelated note: I'd love more command line/GUI tools to be able to use a separate DNS resolver, work on separate interfaces, ...!), and so on. Even if I will never use this guy's init system, if it is even a bit successful it will mean that people will stop assuming systemd, leading to a looser coupling, more interface stability, easier porting between systems, and more choice. So I think it's a great thing he is trying.
- lima 9y ago
  > I don't need socket activation. I don't need a separate DNS resolver in my init system It's. Not. No idea why people keep repeating this. The DNS resolver belongs to systemd-resolved, is entirely optional and off by default. Nothing about it is in the init system. And maybe you don't need socket activation, but there are plenty of valid use cases for it. See: http://0pointer.de/blog/projects/socket-activation.html http://0pointer.de/blog/projects/socket-activation.html
  - mook 9y ago
    "systemd" is the init system, therefore anything that has it in the name is part of that init system. See also journald, which nobody mentions as being part of the init system even though it's more tied to it (is it even possible to have systemd without journald?) If they want to change that perception, they will need to change the name (and hopefully stick a stable ABI around it, but that's unlikely to ever happen).
lmm 9y ago
> Am I the exception? Can't be so, if most linux distros have adopted systemd. By that logic why use Linux in the first place when Windows is so much more popular? If it's anything like other projects from the same source systemd is great when it works and impossible to debug when it doesn't. Which is exactly what I came to OSS to get away from.
phil21 9y ago
The basic summary is basically: Developers love systemd, it's much more "developer friendly" and the way it's laid out makes sense to them (e.g. your love of journalctl, which is bizzaro-world for me). No more thinking about ugly operating systems and those silly details! Old school sysadmins and those who actually get their hands dirty with Linux systems under the hood despise it for what it is. It's the windowsfication of Linux, in my - and many others - opinions. A giant PID 1 that controls everything, re-implementing such basics as ntp and rm, is imo entirely anti-unix-like. Systemd is great until it breaks. Once it breaks though, good luck. They've re-invented the wheel multiple times over, and as time goes on more and more things become part of monolithic systemd. If you do happen to find a weird bug, prepare to get yelled at and be forced to vigorously defend your bug report. The joke in the office is when Pulseaudio support gets added.
- mateuszf 9y ago
  > A giant PID 1 that controls everything Kernel is also a giant code that controls everything. Is this such a bad thing?
  - VLM 9y ago
    Yes. There's been a lot of theoretical and practical work around the last couple decades about putting as little as possible in your single point of technical and legal/patent failure. Small tools that do as little as possible, but do it very well, composed into bigger systems. Philosophically identical to RISC vs CISC.
  - Twirrim 9y ago
    The kernel is a component that has a straightforward purpose on a system, and interferes as little as possible with the actual usage of the system. You could describe it as "It gets the job done, and stays the hell out of the way" Even with the kernel there is an element of choice about what parts of it you do / don't use, e.g. you can use ext4, XFS etc. etc. and even have the choice of providing kernel modules so you can bolt on ZFS. The kernel doesn't start to dictate who or what software does DNS resolution, for example. systemd keeps taking over more and more control of various aspects of your system and effectively removing choice by making it hard to use alternatives because it starts to make assumptions based on the inbuilt versions. systemd is presenting you with a "one-size-fits-all" scenario. The problem is one size never fits all. One of the biggest strengths within the *nix ecosystem has been the fundamental understanding no two use cases are entirely the same, what works well for one purpose doesn't work well for others, and choosing the right software to meet those various purposes is invaluable. It's impossible to make software that is efficient for every single use case. The freedom and possibility to choose the right tool for the right task is essential to designing and building new, innovative, or even just plain old efficient systems.
dwc 9y ago
> abandoning systemd to build another... systemd Well, systemd is a particular answer to a set of problems that have been well known for a while. Building an alternative solutions to those problems is not necessarily "another systemd". > drama surrounding systemd The quick summary: many people are critical of the architecture and design choices and implementation details in systemd in particular, rather than that the underlying issues it's supposed to address are real. > but I've liked using [systemd] so far This isn't surprising, as some of the problems it addresses are real, and a pain. But there are other implementations that also address many of the same issues. Whether those would make you as happy as systemd is another matter.

by atonse 9y ago

ragebol 9y ago

> [systemd] easily the most attractive feature in Ubuntu 16.04 LTS. For me it is as well, I really miss it when working with a Ubuntu 14.04 box. I have not found a single downside to systemd, really.

usernam 9y ago

systemd is not perfect. Debugging systemd is still very hard. There is no way to simulate what actions would be performed to reach a target without actually executing them. There are plenty of issues related to boot and shutdown which are _hard_ to track down because of this. The journal is nice, especially on desktop systems, but it's way too slow to be used in any moderately busy system. The disk space utilization is also _prohibitive_ for archival. We still rely on remote syslog because of this, but it's not without issues: the journal->syslog translation is lossy, and it's compounded with some "quirky" upstream decisions (purely cosmetic) to write unit descriptions in the log as opposed to the real unit name which make tracking down units harder if you don't have the actual journal file. Journal files also get corrupted easily with unclean shutdown. In case of issues (from software to hardware), the local journal quickly becomes completely useless. All in all, systemd is an improvement, but it lacks the debugging tools, performance and overall polish you would expect from a central piece of software such as init. The internal APIs are mostly undocumented or self-referential. The DBus interface is opaque. Things gets muddy very fast as soon as you touch non-central components such as systemd-resolved or networkd. I'm not a detractor, but there are many things systemd can still improve, but it feels we're kind of stuck. I'm quite happy if we have some competition here.

majewsky 9y ago
> Journal files also get corrupted easily with unclean shutdown. Huh? I thought that they introduced this weird binary log format precisely to avoid this kind of error. (I don't remember the exact reasoning, though.)
- usernam 9y ago
  You're confusing the binary format with the tamper protection, or file "sealing". The two features are really independent.
- marcosdumay 9y ago
  "They" were not very successful in this, like they weren't in any other stability related feature.
cmurf 9y ago
The journal files can get corrupted, but it's a localized corruption that journalctl skips over and continues on without additional data loss. Obviously anything that didn't completely commit is gone, which would be gone anyway with some other format. The main difference is that systemd-journald can a.) become aware of the corruption, b.) warn of it rather than display corrupt info, and c.) recover and continue on. There have been a pile of bugs I've run into and reported where journald gets really confused; e.g. 'journalctl -b-5' won't actually go back five boots, it'll show some other content, or it'll give an error, but using --since will work fine. I haven't seen this problem recently, so the fix may be trickling out.
- bryanlarsen 9y ago
  That's not my experience -- once corrupted it stays corrupted. Is this something that's been fixed since Ubuntu 16.04?
  - cmurf 9y ago
    I don't know what you're contradicting because I never said the corruption gets fixed. I said it's skipped over. Specifically what happens is upon detection of corruption, journald stops writing to the corrupt journal file, creates a new journal file, and new entries go there. The utility for reading the journal files, journalctl, can also detect this corruption and will skip over detected corruption and continue on. I have no idea what version of systemd Ubuntu 16.04 has, but in systemd-233 I'm not seeing aforementioned problems with -b switch and corrupt journal files.

gbromios 9y ago

>I get the impression that OP seems to just be abandoning systemd to build another... systemd. So it seems like not-invented-here syndrome. well, he does say: >There may be a bit of NIH syndrome leading to this decision; that’s ok, I can live with that; >I don't really know about any of the drama surrounding systemd I have only been a passive observer of said drama, it boils down to A) init was a stable, widely used, and well understood system that didn't need an update B) systemd takes on far too much responsibility, contrary to the unix philosophy. personally, I have experienced only minor annoyance when having to re-learn familiar commands, though there seems to be a init compatibility layer that has made it pretty transparent to me. It'd probably have been more annoying if I were still an admin.

lmz 9y ago
> init was a stable, widely used, and well understood system that didn't need an update If it didn't need an update why did various parties try and replace it e.g. OSX launchd, Solaris SMF, Ubuntu Upstart?
- gbromios 9y ago
  I have no idea. Like I said, I was only a casual observer of the rally against systemd. I get the feeling that the opposition was more about ethos than any irrefutable technical argument.
  - briffle 9y ago
    Init also took a VERY long time to start, and had very, very rudimentary concepts of dependencies.
- petre 9y ago
  Launchd is actually quite a hurdle to use. XML service files, inconsistent syntax for launchctl commands etc. Apple probably designed it so .plist files would be editable with some GUI tool.

stuaxo 9y ago

I was building a system on raspberry pi involving a few daemons + things that needed starting on powerup + 24h after powerup + before powerdown. It made all these things really easy, I'm not sure why everyone is so keen on doing everything with various bash scripts like we used to.

JdeBP 9y ago
You are making the oft-made error of thinking that System 5 rc was the precursor to systemd. On several major systems it was not. The precursor was upstart on Fedora and Ubuntu, for example. See http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ for the System 5 rc fallacy and several others to avoid.

vidarh 9y ago

I'm sure there's room for reinventing a better systemd, but most of the attempts I've seen either fail to understand and intentionally ignore just how many things systemd simplify. I can understand wanting to decouple and simplify that, but I think a lot of the time the people trying to reinvent it do not understand the concerns that systemd solves for people. E.g. a typical issue is process restarting. Simple. Can be done in a couple of dozen lines. Except that solves nothing for most people: If your server is spinning in place trying to start your big-proper-process-monitor, and it's failing to log anything because rsyslogd or whatever is also just spinning in place, and the boot processes that should have mounted the filesystems needed to get things working is repeatedly crashing? You can mitigate all of that, but now you're creating a separate mechanism (maybe as simple as writing to disk, but still) for early-stage logging, and a separate mechanism and your init needs to deal with things like throttling restart and handling them and so does a separate process manager, and so on. In order to simplify things, there's duplication of functionality all over the place. A huge part of the architecture of systemd is down to starting to pull at that thread and untangling all of these issues. Is it perfect? No. Is it the solution to all of the concerns they've uncovered? Probably not. But what we had before was so much worse, and I'm so happy someone have had a go at addressing them. And frankly, it's great that animosity towards systemd has motivated so many people to finally try to address this issues in other ways. Maybe one of them eventually solves what none of them did before systemd forced peoples hand.

captainmuon 9y ago
I think for many people who are dissatisfied or angry, systemd didn't simplify things. Many people have simple use cases, and for them it made things much more complicated. Monolithic, incomprehensible, unintrospectable. It is like a modern car engine which is just a sealed black box under the hood, full of electronics. Sure, it works great, it is progress compared to before, everything is there for a reason. But you can't easily fix things yourself. And for many people, the old kind of engine was good enough. I mean, for my simple uses, what is wrong with `sudo service nginx restart`, or `sudo killall nginx`? For many people, it is really dead simple. I get that it is not simple for everybody, but that is why we should have choice, not one-size-fits-all.

captainmuon 9y ago

That's fine to me. Linux is about choice. One of my biggest problems with systemd is that other stuff starts to depend on it. For things like GNOME, it is not just an init system, it is THE one init system. Also, I'm sure journald is useful, but I've had so many problems with it. For me as a Desktop and occasional remote computing, small server user, logs are ephemeral. Something goes wrong, I can quickly grep the logs which are plain text. I didn't really feel any advantage from the modern, binary, structured logs. Also, I spent hours installing something on WSL (Linux on Windows), finding that everything works except for the systemd logs, which are empty. The command that controls services spits out hundreds of cryptic lines, services interweaved with all kinds of other objects, when I just want to list the running services. It has too many moving parts for my taste. I don't need socket activation. I don't need a separate DNS resolver in my init system (unrelated note: I'd love more command line/GUI tools to be able to use a separate DNS resolver, work on separate interfaces, ...!), and so on. Even if I will never use this guy's init system, if it is even a bit successful it will mean that people will stop assuming systemd, leading to a looser coupling, more interface stability, easier porting between systems, and more choice. So I think it's a great thing he is trying.

lima 9y ago
> I don't need socket activation. I don't need a separate DNS resolver in my init system It's. Not. No idea why people keep repeating this. The DNS resolver belongs to systemd-resolved, is entirely optional and off by default. Nothing about it is in the init system. And maybe you don't need socket activation, but there are plenty of valid use cases for it. See: http://0pointer.de/blog/projects/socket-activation.html http://0pointer.de/blog/projects/socket-activation.html
- mook 9y ago
  "systemd" is the init system, therefore anything that has it in the name is part of that init system. See also journald, which nobody mentions as being part of the init system even though it's more tied to it (is it even possible to have systemd without journald?) If they want to change that perception, they will need to change the name (and hopefully stick a stable ABI around it, but that's unlikely to ever happen).

lmm 9y ago

> Am I the exception? Can't be so, if most linux distros have adopted systemd. By that logic why use Linux in the first place when Windows is so much more popular? If it's anything like other projects from the same source systemd is great when it works and impossible to debug when it doesn't. Which is exactly what I came to OSS to get away from.

phil21 9y ago

The basic summary is basically: Developers love systemd, it's much more "developer friendly" and the way it's laid out makes sense to them (e.g. your love of journalctl, which is bizzaro-world for me). No more thinking about ugly operating systems and those silly details! Old school sysadmins and those who actually get their hands dirty with Linux systems under the hood despise it for what it is. It's the windowsfication of Linux, in my - and many others - opinions. A giant PID 1 that controls everything, re-implementing such basics as ntp and rm, is imo entirely anti-unix-like. Systemd is great until it breaks. Once it breaks though, good luck. They've re-invented the wheel multiple times over, and as time goes on more and more things become part of monolithic systemd. If you do happen to find a weird bug, prepare to get yelled at and be forced to vigorously defend your bug report. The joke in the office is when Pulseaudio support gets added.

mateuszf 9y ago
> A giant PID 1 that controls everything Kernel is also a giant code that controls everything. Is this such a bad thing?
- VLM 9y ago
  Yes. There's been a lot of theoretical and practical work around the last couple decades about putting as little as possible in your single point of technical and legal/patent failure. Small tools that do as little as possible, but do it very well, composed into bigger systems. Philosophically identical to RISC vs CISC.
- Twirrim 9y ago
  The kernel is a component that has a straightforward purpose on a system, and interferes as little as possible with the actual usage of the system. You could describe it as "It gets the job done, and stays the hell out of the way" Even with the kernel there is an element of choice about what parts of it you do / don't use, e.g. you can use ext4, XFS etc. etc. and even have the choice of providing kernel modules so you can bolt on ZFS. The kernel doesn't start to dictate who or what software does DNS resolution, for example. systemd keeps taking over more and more control of various aspects of your system and effectively removing choice by making it hard to use alternatives because it starts to make assumptions based on the inbuilt versions. systemd is presenting you with a "one-size-fits-all" scenario. The problem is one size never fits all. One of the biggest strengths within the *nix ecosystem has been the fundamental understanding no two use cases are entirely the same, what works well for one purpose doesn't work well for others, and choosing the right software to meet those various purposes is invaluable. It's impossible to make software that is efficient for every single use case. The freedom and possibility to choose the right tool for the right task is essential to designing and building new, innovative, or even just plain old efficient systems.

dwc 9y ago

> abandoning systemd to build another... systemd Well, systemd is a particular answer to a set of problems that have been well known for a while. Building an alternative solutions to those problems is not necessarily "another systemd". > drama surrounding systemd The quick summary: many people are critical of the architecture and design choices and implementation details in systemd in particular, rather than that the underlying issues it's supposed to address are real. > but I've liked using [systemd] so far This isn't surprising, as some of the problems it addresses are real, and a pain. But there are other implementations that also address many of the same issues. Whether those would make you as happy as systemd is another matter.