5 ms·

There is something epically beautiful about OpenBSD's consistency over the last 10 years. I swear that a 2004 OpenBSD sysadmin would be close to 100% efficient

There is something epically beautiful about OpenBSD's consistency over the last 10 years. I swear that a 2004 OpenBSD sysadmin would be close to 100% efficient with a circa 2015 OpenBSD system, despite the absolutely massive number of improvements and changes that have been made with the operating system. Once systemd starts to get traction, it's cognitive overhead is going to become a real problem for sysadmins who want to be able to model the entire boot process and run-state in their head. For better or worse (I honestly don't know which, btw), you are going to have to start to trust the operating system to manage its boot process/run-state correctly rather than having it be a direct implementation of your mental model. (I would argue other *nix Operating systems, like OS X, have, to some degree, been doing that successfully for a while, so it's certainly possible - just not clear whether you want that type of behavior on your server systems. I guess time will tell.)

stephen_g 11y ago
I'm not a professional sysadmin, but as a developer with a bunch of virtual servers systemd has only made things far easier. Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works... Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.
- empressplay 11y ago
  ...and that's awesome if you don't need 99.9% uptime and thus need to have absolute faith in your system's stability. However, for mission-critical applications that do it's hard to argue that systemd is a better solution than writing all your scripts by hand. Unless you want to be able to blame systemd if it all goes pear-shaped but I doubt your boss will buy it =)
  - bad_user 11y ago
    I would argue that features such as having explicit dependencies are absolutely required for 99.9% uptime. Building systems resilient to failures also means being able to predictably spin up new instances that you can trust. I'm also a developer and for example one problem I had with my servers on AWS was that my process sometimes happened to start before my volume meant for logs was mounted, which meant that all logs were going to my primary volume, with the server crashing at some point due to out of space errors. Another time I was using a local database meant for local caching and again, sometimes it would fully initialize only after my main process was started. And because I was using ELB with auto-scaling, it meant that these incorrectly started servers would be connected to the load-balancer, because the main process was suddenly responding to /ping, even though it wasn't ready yet. And then it would receive thousands of requests per second per server, at a moment in time in which the cluster badly needed fresh instances, which led to an unstable cluster. But only sometimes ;-)
- anko 11y ago
  > Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works... This. I'm not saying systemd is the best answer, but I really think sysadmins could learn a thing or two from developers. Nearly every custom init script I've ever seen has bugs, and isn't portable between systems. Where are the unit tests? Code reuse? Focus on maintainability/readability? Shell scripting used to be amazing, but as a language these days it's behind the times.
  - thaumaturgy 11y ago
    Sysadmins would say that developers could learn a lot from them. Like, documentation, maintaining backwards compatibility, sane upgrade processes ... Don't throw stones in glass houses and all that. We all could stand to learn things from other people. Sure, sysadmins could stand to be better programmers, but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem, so for them, certain best practices in software development aren't all that attractive.
    - EmanueleAina 11y ago
      > but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem Which is exactly why developers don't get to write any documentation other than the bare minimum: software for the client is just a tool, once it is running they want developers to move on to the next problem, not wasting time on any additional documentation (which, arguably, is not strictly a developer task, more a QA one).
  - hedwall 11y ago
    >I'm not saying systemd is the best answer, but I really think >sysadmins could learn a thing or two from developers. >Nearly every custom init script I've ever seen has bugs, and >isn't portable between systems. Nearly every custom init script is written by a developer for the product you desperately have to run because development wanted it, not bothering to give us sys admins time to look into the: packaging, configurability, operability and if a custom module/cookbook/playbook needs to be written. >Where are the unit tests? Code reuse? >Focus on maintainability/readability? If we were given the time, it would be there. My deadline for almost all work I do is "yesterday".
- pnathan 11y ago
  > Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works... I've always found init scripts straightforward to read and write, even though I don't develop them often. I don't care for the upstart/systemd/OSX approaches. I don't want to digest another manual to do what I can do in bash easily.
  - EmanueleAina 11y ago
    If your needs are simple, the resulting unit file is trivial and you can basically copy and paste from any example without looking at any documentation. If your needs are complex, you have better chance by mixing different options in the completely declarative syntax of systemd than mixing shell snippets from different scripts using different styles. In any case systemd happily launches LSB-compatible init scripts, you literally don't lose anything.
    - pnathan 11y ago
      Complex needs are better solved with source code rather than special blobs of $system. > In any case syst emd happily launches LSB-compatible init scripts, you literally don't lose anything. Yes, I do. I literally have piles of unwanted systemd on my system.
      - EmanueleAina 11y ago
        > Complex needs are better solved with source code rather than special blobs of $system. Arguably, reinventing multiple complex wheels is often a not particularly good choice. So either you're implementing something really novel in your init script (which does not sound like the best place for it) or it's something that may benefit from a single, shared and tested implementation. And, again, you can still fallback to LSB shell scripts if you really need something incredibly special: my point was that, bloat notwithstanding, you don't lose any feature but gain many.
- yellowapple 11y ago
  Professional sysadmin (and developer) here :) > Writing init scripts used to be a real pain In the GNU/Linux world, yep, this is the case. Lots of copypasted boilerplate that becomes a nightmare to debug. In the BSD world (at least with OpenBSD, where I'm most experienced), this isn't the case at all; OpenBSD ships with /etc/rc.d/rc.subr, which one can (and should) source into one's initscripts to make life much easier than even (in my experience) systemd's unit files. Set the $daemon variable, source in rc.subr, call rc_cmd, done. Sure, you might want/need to read a couple manpages, but this is the case with systemd, too. Back to the GNU/Linux world, the complexity of SysVInit is not a hard dependency of using shell scripts for daemon management; there's very little stopping a GNU/Linux system from adopting a BSD-style rc system, rc.subr, etc. and making life significantly easier for sysadmins. Alternately, there's very little stopping a GNU/Linux system from implementing shell scripts in something significantly less horrendous to use, such as an interpreted scripting language (like Perl or Python or Ruby). SysVInit and BSD-style rc-based init both have a distinct lack of hard requirements on how initscripts are implemented, so long as said scripts are valid executables. > Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving. Would you not want to be able to model the startup process and run-state of the software you write in your head? Why should the underlying operating system be any different? If you have to rely upon external tools just to be able to reason about the execution flow of something - your server, your program, your recipe for lamb vindaloo, whatever - chances are it's way too complicated and needs significant refactoring.
  - digi_owl 11y ago
    At this point in time i fear that what RH decides to do, everyone else interested in deploying Linux commercially ends up doing. And because of that, developers then target RH related choices as a baseline, further exasperating the inbreeding.
empressplay 11y ago
Except that there's no reason for systemd's model to "get traction" outside of desktop-focussed Linux distributions, since a "better user experience" is it's current raison d'etre. So no, I shouldn't need to have to "learn to trust the operating system / systemd" on my servers -- it has no reason to be there!
- justincormack 11y ago
  The model where services do not daemonize and write a pid to a file is actually hugely better for servers. Most of the rest of it is not useful.
  - EmanueleAina 11y ago
    The per-unit cgroup and the ability to kill it is rather nice too, without delving in the container-related features where the systemd server-side support really shines.
  - JdeBP 11y ago
    The model where services do not daemonize and write a pid to a file is ... ... the one that people who moved away from System 5 rc scripts years ago have been pushing for with some degree of success, for years. systemd is not its source at all. systemd has merely been reaping the benefit of all of the programs over the past 18 years that have gained a --no-fork or a --foreground or a -D or a -F or whatever option in response to those who wanted to run services properly without bodging things under daemontools, or runit, or s6, or perp, or freedt, or daemontools-encore ...
- EmanueleAina 11y ago
  Many systemd features are explicitly tailored for server workload (eg. resource management), and a particular effort is being put over container-based deployments. CoreOS has been quite successful in basing their own infrastructure on systemd and with rkt they are moving to use systemd even more.

stephen_g 11y ago

I'm not a professional sysadmin, but as a developer with a bunch of virtual servers systemd has only made things far easier. Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works... Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.

empressplay 11y ago
...and that's awesome if you don't need 99.9% uptime and thus need to have absolute faith in your system's stability. However, for mission-critical applications that do it's hard to argue that systemd is a better solution than writing all your scripts by hand. Unless you want to be able to blame systemd if it all goes pear-shaped but I doubt your boss will buy it =)
- bad_user 11y ago
  I would argue that features such as having explicit dependencies are absolutely required for 99.9% uptime. Building systems resilient to failures also means being able to predictably spin up new instances that you can trust. I'm also a developer and for example one problem I had with my servers on AWS was that my process sometimes happened to start before my volume meant for logs was mounted, which meant that all logs were going to my primary volume, with the server crashing at some point due to out of space errors. Another time I was using a local database meant for local caching and again, sometimes it would fully initialize only after my main process was started. And because I was using ELB with auto-scaling, it meant that these incorrectly started servers would be connected to the load-balancer, because the main process was suddenly responding to /ping, even though it wasn't ready yet. And then it would receive thousands of requests per second per server, at a moment in time in which the cluster badly needed fresh instances, which led to an unstable cluster. But only sometimes ;-)
anko 11y ago
> Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works... This. I'm not saying systemd is the best answer, but I really think sysadmins could learn a thing or two from developers. Nearly every custom init script I've ever seen has bugs, and isn't portable between systems. Where are the unit tests? Code reuse? Focus on maintainability/readability? Shell scripting used to be amazing, but as a language these days it's behind the times.
- thaumaturgy 11y ago
  Sysadmins would say that developers could learn a lot from them. Like, documentation, maintaining backwards compatibility, sane upgrade processes ... Don't throw stones in glass houses and all that. We all could stand to learn things from other people. Sure, sysadmins could stand to be better programmers, but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem, so for them, certain best practices in software development aren't all that attractive.
  - EmanueleAina 11y ago
    > but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem Which is exactly why developers don't get to write any documentation other than the bare minimum: software for the client is just a tool, once it is running they want developers to move on to the next problem, not wasting time on any additional documentation (which, arguably, is not strictly a developer task, more a QA one).
- hedwall 11y ago
  >I'm not saying systemd is the best answer, but I really think >sysadmins could learn a thing or two from developers. >Nearly every custom init script I've ever seen has bugs, and >isn't portable between systems. Nearly every custom init script is written by a developer for the product you desperately have to run because development wanted it, not bothering to give us sys admins time to look into the: packaging, configurability, operability and if a custom module/cookbook/playbook needs to be written. >Where are the unit tests? Code reuse? >Focus on maintainability/readability? If we were given the time, it would be there. My deadline for almost all work I do is "yesterday".
pnathan 11y ago
> Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works... I've always found init scripts straightforward to read and write, even though I don't develop them often. I don't care for the upstart/systemd/OSX approaches. I don't want to digest another manual to do what I can do in bash easily.
- EmanueleAina 11y ago
  If your needs are simple, the resulting unit file is trivial and you can basically copy and paste from any example without looking at any documentation. If your needs are complex, you have better chance by mixing different options in the completely declarative syntax of systemd than mixing shell snippets from different scripts using different styles. In any case systemd happily launches LSB-compatible init scripts, you literally don't lose anything.
  - pnathan 11y ago
    Complex needs are better solved with source code rather than special blobs of $system. > In any case syst emd happily launches LSB-compatible init scripts, you literally don't lose anything. Yes, I do. I literally have piles of unwanted systemd on my system.
    - EmanueleAina 11y ago
      > Complex needs are better solved with source code rather than special blobs of $system. Arguably, reinventing multiple complex wheels is often a not particularly good choice. So either you're implementing something really novel in your init script (which does not sound like the best place for it) or it's something that may benefit from a single, shared and tested implementation. And, again, you can still fallback to LSB shell scripts if you really need something incredibly special: my point was that, bloat notwithstanding, you don't lose any feature but gain many.
yellowapple 11y ago
Professional sysadmin (and developer) here :) > Writing init scripts used to be a real pain In the GNU/Linux world, yep, this is the case. Lots of copypasted boilerplate that becomes a nightmare to debug. In the BSD world (at least with OpenBSD, where I'm most experienced), this isn't the case at all; OpenBSD ships with /etc/rc.d/rc.subr, which one can (and should) source into one's initscripts to make life much easier than even (in my experience) systemd's unit files. Set the $daemon variable, source in rc.subr, call rc_cmd, done. Sure, you might want/need to read a couple manpages, but this is the case with systemd, too. Back to the GNU/Linux world, the complexity of SysVInit is not a hard dependency of using shell scripts for daemon management; there's very little stopping a GNU/Linux system from adopting a BSD-style rc system, rc.subr, etc. and making life significantly easier for sysadmins. Alternately, there's very little stopping a GNU/Linux system from implementing shell scripts in something significantly less horrendous to use, such as an interpreted scripting language (like Perl or Python or Ruby). SysVInit and BSD-style rc-based init both have a distinct lack of hard requirements on how initscripts are implemented, so long as said scripts are valid executables. > Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving. Would you not want to be able to model the startup process and run-state of the software you write in your head? Why should the underlying operating system be any different? If you have to rely upon external tools just to be able to reason about the execution flow of something - your server, your program, your recipe for lamb vindaloo, whatever - chances are it's way too complicated and needs significant refactoring.
- digi_owl 11y ago
  At this point in time i fear that what RH decides to do, everyone else interested in deploying Linux commercially ends up doing. And because of that, developers then target RH related choices as a baseline, further exasperating the inbreeding.

empressplay 11y ago

Except that there's no reason for systemd's model to "get traction" outside of desktop-focussed Linux distributions, since a "better user experience" is it's current raison d'etre. So no, I shouldn't need to have to "learn to trust the operating system / systemd" on my servers -- it has no reason to be there!

justincormack 11y ago
The model where services do not daemonize and write a pid to a file is actually hugely better for servers. Most of the rest of it is not useful.
- EmanueleAina 11y ago
  The per-unit cgroup and the ability to kill it is rather nice too, without delving in the container-related features where the systemd server-side support really shines.
- JdeBP 11y ago
  The model where services do not daemonize and write a pid to a file is ... ... the one that people who moved away from System 5 rc scripts years ago have been pushing for with some degree of success, for years. systemd is not its source at all. systemd has merely been reaping the benefit of all of the programs over the past 18 years that have gained a --no-fork or a --foreground or a -D or a -F or whatever option in response to those who wanted to run services properly without bodging things under daemontools, or runit, or s6, or perp, or freedt, or daemontools-encore ...
EmanueleAina 11y ago
Many systemd features are explicitly tailored for server workload (eg. resource management), and a particular effort is being put over container-based deployments. CoreOS has been quite successful in basing their own infrastructure on systemd and with rkt they are moving to use systemd even more.