4 ms·

> It can be argued that Systemd does do just one thing... Systemd is a process lifecycle supervisor. It tracks the lifetimes of processes. That's about it. It'

> It can be argued that Systemd does do just one thing... Systemd is a process lifecycle supervisor. It tracks the lifetimes of processes. That's about it. It's also a /dev node manager, and a syslogd replacement, and a cron replacement, and a dhcpcd replacement, and a mount/umount, insmod/modprobe reimplementation, [0] and a bootloader, [1] and... The systemd project is huge, sprawling, and continues to grow. > Cron is probably the worst example of "do one thing well", because it does that one thing very poorly. It can only start stuff. [If you want something sophisticated you inevitably have to implement systemd.] It turns out that you don't! Check out fcron. Its crontab is described here: [2]. If you're impatient, search for "The options can be set either for every line". > Things built on top of [Unix's process model] tend to be shell scripts. The phrase "shell scripts" gets tossed around like it's a slur. What are the essential differences between a Bash program, a Python program, a TCL program, an Erlang program, and a C program that all do the very same thing? What makes "shell scripts" so undesirable? [0] Given the number of times things like recursive bind umounts, module loading/unloading, and more core stuff have broken on systemd-enabled systems and only such systems, it's almost impossible to believe that mount/umount, insmod/modprobe & etc. aren't being reimplemented in the systemd project. ;) [1] https://wiki.archlinux.org/index.php/Systemd-boot https://wiki.archlinux.org/index.php/Systemd-boot [2] http://fcron.free.fr/doc/en/fcrontab.5.html http://fcron.free.fr/doc/en/fcrontab.5.html

atombender 11y ago
> Check out fcron ... Well. * Can it let me start jobs manually? * Can I defer their scheduling programmatically without editing a file to comment it out? * Can I add or remote new jobs programmatically and atomically? * Can I ask it about the status of a job? * If a job fails, is it retried, up to a limit? * Can I give it a time budget after which the job is killed? * Can I give it a resource budget (e.g. CPU usage)? * Can it log the output tagged with an identifier that identifies the job uniquely across time? * Can it notify me by some programmatic mechanism (so that I can pipe it to a webhook, say) if a job fails? * Can I programmatically start jobs, edit jobs, etc. from a different host? * Can I ensure that only a single instance of the job runs in a cluster, if multiple hosts have been configured with the same job? * If the job runs so long that it overlaps with the next scheduled period, can it ensure that the next job doesn't start yet? * If the job forks, can it ensure that the children are cleaned up? These are some of the features that I want from a job scheduler. Yes, some of it can be cobbled together with wrapper scripts, manual cgroups, and so on. But I don't want to write all of that. > What are the essential differences between a Bash program Shell scripts are brittle. That's hardly debatable. You can write solid shell scripts, but people generally don't because it takes a lot of effort and arcane knowledge to do it successfully. A lot of people don't know about classic safeguards such as "-e", "-o pipefail", exit traps and so on. Some of the safeguards you want are hard to do safely and atomically. Unix has a nice pipe system, but shell scripts that need to drive a lot of things invariably mess it up because everything pipes to the same place. You get errors and information output intermingled and there's often no way to tell who emitted it without going step by step. Throughout my 20+ years of using Unix variants, shell scripts and "shell-scripty interdependencies" have always been a source of problems, usually related to things like escaping, quoting, option parsing, alias expansion, variable expansion, bad toolchain detection (e.g. assuming your "ls" is GNU), ignoring exit codes, bad output parsing, etc. "find | xargs" works just fine until the day you encounter a file name containing spaces. Then you do "find -print0 | xargs -0" and you're happy until the next edge case comes along that touches on space handling, of which the shell language family is rife. It gets worse when shell scripts try to be nice over SSH. At least languages like Python and Erlang have first-class support for primitives, even if their dynamic typing leads to a lot of potential situations where the runtime parameters aren't what the author assumed, and things blow up because someone didn't sufficiently duck-type. I'm not a huge fan of Go, but Go is a surprisingly good fit for command-line tools, and its strictness and static typing helps avoid a whole class of errors. I love shell scripts, and I write them nearly every day, but I would never build an OS on top of them.
- simoncion 11y ago
  Several of the items in your bulleted list make it seem like you either didn't read all of, or didn't thoroughly understand fcron's crontab manpage. I urge you to go back and re-read the documentation, or -at least- the Options section. I'll address the items that aren't covered by the documentation that I linked to. :) > * Can it let me start jobs manually? > * Can I ask it about the status of a job? Yep. [0] > * Can I defer their scheduling programmatically without editing a file to comment it out? I'm not sure what it means to "defer" scheduling. That sounds like changing the time a job starts. If I'm wrong about that, please do let me know. :) Anyway. Every job scheduling system stores its non-volatile info on disk. fcron's full-featured interface to its job scheduling system is found in [1]. I don't know if that meets your requirement. > * Can I programmatically start jobs, edit jobs, etc. from a different host? Yep. [0] [1] > * Can I add or [remove] new jobs programmatically and atomically? AFAICT, yes. [1] > * Can it log the output tagged with an identifier that identifies the job uniquely across time? > * Can it notify me by some programmatic mechanism (so that I can pipe it to a webhook, say) if a job fails? Yep. That's in the email that it sends. What your reporting infrastructure does with that email is up to you. > * Can I ensure that only a single instance of the job runs in a cluster, if multiple hosts have been configured with the same job? AFAIK, Systemd doesn't do this, so I don't know why you're asking for it. (Other than the fact that it would be rather nice to have.) :) But, no. AFAICT, fcron runs on a single host. I have heard about chronos, though. [2] So, I need to preface the rest of this commentary with the following: Bash is clunky, kinda unwieldy, and very, very far from my favorite language. There are several reasons why I select Python for scripts that get much more complex than "rather simple". Please keep this fact in mind while reading the rest of my commentary. :) > Shell scripts are brittle. ... You can write solid shell scripts, but people generally don't because it takes a lot of effort and arcane knowledge to do it successfully. But C programs written by people who can't be arsed are also brittle and -even worse- the language itself is filled with hidden and subtle pitfalls! > Unix has a nice pipe system, but shell scripts that need to drive a lot of things invariably mess it up because everything pipes to the same place [and this makes unwinding intermingled output difficult sometimes]. So, why not religiously do something like using logger(1) with appropriate tags and ids? I mean, you actually have to work a little to get reasonable logging in (almost?) every language. Hell, -IIRC- C doesn't ship with anything substantially more sophisticated than printf(3) and friends. ;) > Then you do "find -print0 | xargs -0" and you're happy until the next edge case comes along that touches on space handling... How does IFS=$'\n' and/or the "${VAR}" pattern not fix all space handling problems when using bash? (Bash is one of my weaker languages, so if you know, I'm genuinely interested in hearing about it.) However, if your answer is something like "Some other script you use might screw things up!", then my reply is "Bash and sh programs are not the only languages in which we might find an erroneous program on which we depend.". :) > It gets worse when shell scripts try to be nice over SSH. How's that? I'm seriously asking, for my own edification. > I love shell scripts, and I write them nearly every day, but I would never build an OS on top of them. I don't think that I (or anyone else who has an informed opinion on the topic of sysvrc replacements) has ever seriously suggested that one built an entire OS on top of shell scripts. Frankly, they're insufficiently powerful (not to mention [comparatively speaking] too damn slow) to reasonably accomplish the task. However, OpenRC -and so many other RC and init systems- demonstrate that shell scripts can provide rather powerful, relatively foolproof methods for a packager or service author to control services on their system. > At least ... Erlang [has] first-class support for primitives, even if their dynamic typing leads to a lot of potential situations where the runtime parameters aren't what the author assumed, and things blow up because someone didn't sufficiently duck-type. I am totally, seriously, not (!!) getting on your case here, but how much work have you done with Erlang? I'm a novice-to-middling Erlang programmer, and the situation you described seems... difficult to run into unless you're unusually careless. (Python, OTOH...) > I'm not a huge fan of Go... Me either! It's so opinionated. Does it still consider unused imports to be compile-stopping errors? :( [0] http://fcron.free.fr/doc/en/fcrondyn.1.html http://fcron.free.fr/doc/en/fcrondyn.1.html [1] http://fcron.free.fr/doc/en/fcrontab.1.html http://fcron.free.fr/doc/en/fcrontab.1.html [2] https://aphyr.com/posts/326-call-me-maybe-chronos https://aphyr.com/posts/326-call-me-maybe-chronos

atombender 11y ago

> Check out fcron ... Well. * Can it let me start jobs manually? * Can I defer their scheduling programmatically without editing a file to comment it out? * Can I add or remote new jobs programmatically and atomically? * Can I ask it about the status of a job? * If a job fails, is it retried, up to a limit? * Can I give it a time budget after which the job is killed? * Can I give it a resource budget (e.g. CPU usage)? * Can it log the output tagged with an identifier that identifies the job uniquely across time? * Can it notify me by some programmatic mechanism (so that I can pipe it to a webhook, say) if a job fails? * Can I programmatically start jobs, edit jobs, etc. from a different host? * Can I ensure that only a single instance of the job runs in a cluster, if multiple hosts have been configured with the same job? * If the job runs so long that it overlaps with the next scheduled period, can it ensure that the next job doesn't start yet? * If the job forks, can it ensure that the children are cleaned up? These are some of the features that I want from a job scheduler. Yes, some of it can be cobbled together with wrapper scripts, manual cgroups, and so on. But I don't want to write all of that. > What are the essential differences between a Bash program Shell scripts are brittle. That's hardly debatable. You can write solid shell scripts, but people generally don't because it takes a lot of effort and arcane knowledge to do it successfully. A lot of people don't know about classic safeguards such as "-e", "-o pipefail", exit traps and so on. Some of the safeguards you want are hard to do safely and atomically. Unix has a nice pipe system, but shell scripts that need to drive a lot of things invariably mess it up because everything pipes to the same place. You get errors and information output intermingled and there's often no way to tell who emitted it without going step by step. Throughout my 20+ years of using Unix variants, shell scripts and "shell-scripty interdependencies" have always been a source of problems, usually related to things like escaping, quoting, option parsing, alias expansion, variable expansion, bad toolchain detection (e.g. assuming your "ls" is GNU), ignoring exit codes, bad output parsing, etc. "find | xargs" works just fine until the day you encounter a file name containing spaces. Then you do "find -print0 | xargs -0" and you're happy until the next edge case comes along that touches on space handling, of which the shell language family is rife. It gets worse when shell scripts try to be nice over SSH. At least languages like Python and Erlang have first-class support for primitives, even if their dynamic typing leads to a lot of potential situations where the runtime parameters aren't what the author assumed, and things blow up because someone didn't sufficiently duck-type. I'm not a huge fan of Go, but Go is a surprisingly good fit for command-line tools, and its strictness and static typing helps avoid a whole class of errors. I love shell scripts, and I write them nearly every day, but I would never build an OS on top of them.

simoncion 11y ago
Several of the items in your bulleted list make it seem like you either didn't read all of, or didn't thoroughly understand fcron's crontab manpage. I urge you to go back and re-read the documentation, or -at least- the Options section. I'll address the items that aren't covered by the documentation that I linked to. :) > * Can it let me start jobs manually? > * Can I ask it about the status of a job? Yep. [0] > * Can I defer their scheduling programmatically without editing a file to comment it out? I'm not sure what it means to "defer" scheduling. That sounds like changing the time a job starts. If I'm wrong about that, please do let me know. :) Anyway. Every job scheduling system stores its non-volatile info on disk. fcron's full-featured interface to its job scheduling system is found in [1]. I don't know if that meets your requirement. > * Can I programmatically start jobs, edit jobs, etc. from a different host? Yep. [0] [1] > * Can I add or [remove] new jobs programmatically and atomically? AFAICT, yes. [1] > * Can it log the output tagged with an identifier that identifies the job uniquely across time? > * Can it notify me by some programmatic mechanism (so that I can pipe it to a webhook, say) if a job fails? Yep. That's in the email that it sends. What your reporting infrastructure does with that email is up to you. > * Can I ensure that only a single instance of the job runs in a cluster, if multiple hosts have been configured with the same job? AFAIK, Systemd doesn't do this, so I don't know why you're asking for it. (Other than the fact that it would be rather nice to have.) :) But, no. AFAICT, fcron runs on a single host. I have heard about chronos, though. [2] So, I need to preface the rest of this commentary with the following: Bash is clunky, kinda unwieldy, and very, very far from my favorite language. There are several reasons why I select Python for scripts that get much more complex than "rather simple". Please keep this fact in mind while reading the rest of my commentary. :) > Shell scripts are brittle. ... You can write solid shell scripts, but people generally don't because it takes a lot of effort and arcane knowledge to do it successfully. But C programs written by people who can't be arsed are also brittle and -even worse- the language itself is filled with hidden and subtle pitfalls! > Unix has a nice pipe system, but shell scripts that need to drive a lot of things invariably mess it up because everything pipes to the same place [and this makes unwinding intermingled output difficult sometimes]. So, why not religiously do something like using logger(1) with appropriate tags and ids? I mean, you actually have to work a little to get reasonable logging in (almost?) every language. Hell, -IIRC- C doesn't ship with anything substantially more sophisticated than printf(3) and friends. ;) > Then you do "find -print0 | xargs -0" and you're happy until the next edge case comes along that touches on space handling... How does IFS=$'\n' and/or the "${VAR}" pattern not fix all space handling problems when using bash? (Bash is one of my weaker languages, so if you know, I'm genuinely interested in hearing about it.) However, if your answer is something like "Some other script you use might screw things up!", then my reply is "Bash and sh programs are not the only languages in which we might find an erroneous program on which we depend.". :) > It gets worse when shell scripts try to be nice over SSH. How's that? I'm seriously asking, for my own edification. > I love shell scripts, and I write them nearly every day, but I would never build an OS on top of them. I don't think that I (or anyone else who has an informed opinion on the topic of sysvrc replacements) has ever seriously suggested that one built an entire OS on top of shell scripts. Frankly, they're insufficiently powerful (not to mention [comparatively speaking] too damn slow) to reasonably accomplish the task. However, OpenRC -and so many other RC and init systems- demonstrate that shell scripts can provide rather powerful, relatively foolproof methods for a packager or service author to control services on their system. > At least ... Erlang [has] first-class support for primitives, even if their dynamic typing leads to a lot of potential situations where the runtime parameters aren't what the author assumed, and things blow up because someone didn't sufficiently duck-type. I am totally, seriously, not (!!) getting on your case here, but how much work have you done with Erlang? I'm a novice-to-middling Erlang programmer, and the situation you described seems... difficult to run into unless you're unusually careless. (Python, OTOH...) > I'm not a huge fan of Go... Me either! It's so opinionated. Does it still consider unused imports to be compile-stopping errors? :( [0] http://fcron.free.fr/doc/en/fcrondyn.1.html http://fcron.free.fr/doc/en/fcrondyn.1.html [1] http://fcron.free.fr/doc/en/fcrontab.1.html http://fcron.free.fr/doc/en/fcrontab.1.html [2] https://aphyr.com/posts/326-call-me-maybe-chronos https://aphyr.com/posts/326-call-me-maybe-chronos