20 ms·

Love systemd timers

• ktm5j 4mo ago
  Oh I love them quite a lot! I use them to run all of our backup jobs, easy to set up and have never had an issue.
• jjgreen 4mo ago
  I've been almost convinced by systemd (and have switched to using it), but God the syntax of those service files is so ugly ...
  • WesolyKubeczek 4mo ago
    Could have been worse. Could have been YAML. Could have been XML.
    • silvestrov 4mo ago
      XML would have the advantage of having a grammar so we could validate the config files. It would also make it much simpler to make good GUI editors for the files instead of the Notepad approach most unix config files take.
      • Juliate 4mo ago
        There are good GUI editors for XML?
      • WesolyKubeczek 4mo ago
        Since systemd is successfully parsing its INI files, and barks at you when you put weird shit into them, a grammar for them does exist as well. XML is that wonderful format that gave us vulnerabilities like death by million laughs, up to a certain moment, you could MitM DTDs, and a whole slew of everything-XML stuff back when XML was like AI is today, none of which I miss today. Oh, and remember times when programmers would argue whether argument order in XML files should be significant or not? But XML books with their idealized XML future description did give me the same warm fuzzies as some intricate clockwork mechanism to a Victorian geek.
      • deleted 4mo ago
        [deleted]
      • pwdisswordfishq 4mo ago
        The systemd dialect of INI is actually pretty well-defined though. https://www.freedesktop.org/software/systemd/man/latest/systemd.syntax.html https://www.freedesktop.org/software/systemd/man/latest/syst...
        • silvestrov 4mo ago
          I'd really like a collection of unit tests for parsers. There is a lot of details that can differ between parsers. E.g. in "Section C" the resulting KeyThree is "value 3▵▵▵▵▵▵▵value 3 continued" where each "▵" symbol is a space. I think most people would expect a single (or no) space here. I would guess that most software would strip the comments in SectionC or rearrange the output so that it will result in a diff even when nothing in SectionC changed. So if you edit the file by hand in the same style as shown in the examples, then most editors would not be able to make a minor edit without making a large diff as many sections would be formatted differently.
    • jjgreen 4mo ago
      To be honest, I think either of those would have been better ...
      • WesolyKubeczek 4mo ago
        /me cowers in fear
    • wpm 4mo ago
      Could have been better. Could have been XML Property Lists. ducks
    • Tsiklon 4mo ago
      XML - I see you’ve used macOS’ LaunchD, the system that inspired Systemd
      • WesolyKubeczek 4mo ago
        Yeah, I'm a man of culture like this. However, systemd with its service dependencies runs circles around launchd in pretty much every aspect.
        • Tsiklon 4mo ago
          Service dependency resolution and parallel startup is super in systemd. Big fan of what I can do with it.
  • whateveracct 4mo ago
    This is why I like NixOS. Defining systemd services in it is very neat.
  • zamadatix 4mo ago
    Never thought I'd see hackers saying INI format looked ugly of all things. It's basic, sure, but that's a good thing for something meant to be easily editable by hand from any editor. Otherwise, it's just key value pairs in named sections, how ugly can it be about that?
    • deleted 4mo ago
      [deleted]
    • jjgreen 4mo ago
      key-value pairs where the = cannot be surrounded by spaces, so I have to write [Service] Type=oneshot WorkingDirectory={{ home }}/current/ Environment=RAILS_ENV=production ExecStart=/bin/sh -lc "bin/db-backup --verbose" which fills me with sadness
      • yjftsjthsd-h 4mo ago
        What? You absolutely can have spaces; most of mine look more like [Service] Type = oneshot WorkingDirectory = %h/current/ Environment = RAILS_ENV=production ExecStart = /bin/sh -lc "bin/db-backup --verbose"
        • jjgreen 4mo ago
          Friend, you have changed my life
          • weavie 4mo ago
            Is this one of those cases where at one point you had an error in the file and you figured it was down to spaces? You fixed that issue, it still didn't work but from that point you never thought to question the assumption. I find myself doing this sort of thing all the time..
            • troyvit 4mo ago
  • mrweasel 4mo ago
    There's definitely some weirdness to certain parts of systemd service files, but was a huge improvement over Upstart and the old SysV-style init scripts. Over all I think Systemd get way to much criticism. You don't have to use all the parts, but if you care to go through the documentation you'll find interesting features such as journald log-shipping and systemd-machined which can manage containers and VMs.
    • okanat 4mo ago
      I love how easy it is to create a completely isolated daemons with systemd. In a single .service file one can define a daemon that has a very limited view to the filesystem, can only open specific devices, uses randomized UIDs, and has limited capabilities: https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html https://www.freedesktop.org/software/systemd/man/latest/syst... It is way simpler and cleaner than Docker/Podman IMHO.
  • SEJeff 4mo ago
    Oh yes, because the well documented clean syntax of sys v init shell scripts was so nice. If I never recall hacking in ulimit calls in the top of buggy shell scripts for crappy old services that done respect pam_limits it won’t be soon enough.
  • linsomniac 4mo ago
    Hard disagree. Compared to an init script, with all its boilerplate, I'd take a systemd unit file.
  • egorfine 4mo ago
    As a passionate systemd hater I would say I do not agree. Cron syntax is worse.
  • dominiwe 4mo ago
    Relevant, a golfed systemd polyglot file that is simultaneously an executable script: https://domi.work/blog/posts/compose_polyglot/ https://domi.work/blog/posts/compose_polyglot/ Yes, I have too much time sometimes... and I agree, I don't like the syntax.
• iso1631 4mo ago
  > humble systemd
  • pwdisswordfishq 4mo ago
    That the same cannot be said of its maintainer is another matter.
    • the_real_cher 4mo ago
      The main person in charge of Linux itself isn't considered the most humble but they make amazing products.
      • dijit 4mo ago
        Linus is pretty humble tbh, he just expects that people don't throw shit over the fence.
• deleted 4mo ago
  [deleted]
• hombre_fatal 4mo ago
  NixOS comes with systemd, so I've been using it as a first-class part of managing stuff. It's great, especially coming from macOS' launchd. Which makes it nice to distribute a tool for NixOS so that it can lean into systemd instead of as some bolted-on afterthought. Makes me wonder what you'd do if you were distributing a lifecycle-heavy tool for Linux users in general since systemd isn't ubiquitous. I use a systemd timer to run a monthly scrub for my btrfs pool. Kinda cool how you can do increasingly useful things like skip the next scheduled event if the user initiates a scrub, do or don't accumulate tasks if you have a monthly task but the machine was offline for 6 months -- or fold them into a single task, etc.
  • drunner 4mo ago
    Have you been defining them directly in your flake.nix file? I too am on nixos but I keep all my configurations in their native format and symlink them with nix, that way I can take and reuse that config on a non nixos system easily. The problem I have found is that nixos doesn't seem to pickup and run systemd timers and services placed into the ~/.config/systems/user folder and additionally things like WantedBy=default.target have no effect. So after I restart all my services manually on reboot I agree, systems timers are cool.
    • Cyph0n 4mo ago
      I define all units in Nix because: a) It is way nicer and you get decent validation at build time b) A LLM can port units over if the need arises; it’s a very light abstraction around systemd syntax c) I personally don’t see how I would ever move to another distro :)
      • zaik 4mo ago
        b) Or you could take the compiled units from /etc/systemd and copy them wherever
        • Cyph0n 4mo ago
          I believe that probably won’t work unfortunately; the generated unit files have a bunch of hardcoded Nix store paths.
• iluvcommunism 4mo ago
  [dead]
• andrewstuart 4mo ago
  Even better is systemd socket activation.
  • interf4ce 4mo ago
    This is very interesting. I'm not sure what I'd use it for yet, but I imagine it could be useful for triggering ad hoc jobs over the network. Maybe have Home Assistant make a network call to kick off a daily back up when I leave the office at the end of a work day.
    • kevinmgranger 4mo ago
      I believe its original motivation was just speeding up boot times by starting fewer services, even if you'd eventually want the service running. This was achieved in the past with xinetd, but systemd made the approach more popular for the masses.
      • roryirvine 4mo ago
        inetd began to fall out of favour in the mid-late 90s as services became more heavyweight and startup times became longer (think of the initial crypto setup needed by sshd vs rsh/telnetd) CPU speeds have increased & and i/o latency has decreased so much since then that startup times are generally imperceptible, so the pendulum has swung back to favouring socket activation. The anti-systemd "traditionalists" never seem to acknowledge that history, though!
    • urineaut 4mo ago
      A pretty nice use case I have for socket activation is for isolating containers or applications from the host network. The great thing about socket activation is that opened sockets carry over even if the application/container unshares into a different network namespace! It also works great with Podman pods with networking in the pod completely disabled and, as those are host sockets, does fully retain the connection info of peers (so logs are not just uselessly containing the gateway IP, depending on the container network config)
  • 4mo ago
• gchamonlive 4mo ago
  Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixed time. With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available. Btw this is my repo for the backup automation: https://github.com/gchamon/borg-automated-backups https://github.com/gchamon/borg-automated-backups
  • mid-kid 4mo ago
    Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules. To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this. Many cron implementations have a similar mechanism.
    • gchamonlive 4mo ago
      EDITED This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
      • newsoftheday 4mo ago
        > fallback to running when the system comes online. That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described. I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then. > With @hourly I lose this control and multiple machines could potentially trigger backups at the same time Then don't use @hourly, use staggered times, it's very easy.
• stryan 4mo ago
  Timers can work with arbitrary units (not just a similarly-named service unit) so they can be surprisingly flexible. I have a timer on my servers that starts a backup.target that fires off a full "restic backup","restic prune", "restic forget" backup cycle each morning with randomized start times and notifications. The actual restic-* units are Podman Quadlets so the whole setup runs agnosticaly of what's on the server, just as long as it has Podman and Systemd installed. I will admit thought, timers are up there in terms of being the clunkiest systemd unit type to use on a regular basis. I get why they're split up into two files and require different start vs enable syntax's, but man sometimes I just want to create a file that runs a script and be done with it.
  • esperent 4mo ago
    Why do you randomize your backup times?
    • stryan 4mo ago
      Should have been more clear: I use RandomizedOffsetSec= to add a random offset to a set start time (usually 4am), to prevent overloading the backup server, not truly random start times.
      • capitainenemo 4mo ago
        As someone else noted, that's also a cron feature
        • c-hendricks 4mo ago
          A feature of _some_ cron systems
          • capitainenemo 4mo ago
            True. But commonly used ones.
  • 4mo ago
• lanycrost 4mo ago
  systemd is complex on first view, but after using it you didn't want to use anything else. It's handy to manage everything using systemctl
  • alyandon 4mo ago
    That and systemd having actually useful man pages.
  • the_real_cher 4mo ago
    This is such a modern view. People used to HATE systemd when it first came out, but I always liked it and knew people would eventually come around and its nice to see they finally did!
    • egorfine 4mo ago
      Some people are stubborn and refuse to see how obviously superior systemd is to the old ways. Me included.
    • weaksauce 4mo ago
      the thing for me is I started using the init system and while it was fine it always felt brittle for some reason. systemd feels solid and robust like it was well thought out. maybe i'm off base and didn't know how to use init effectively but it was my feeling. that and cron always felt fragile too with a lot of quirks and limitations you had to work around instead of being a robust thing from the start.
    • eichin 4mo ago
      Arguably they (we :-) were right at the time. Around Ubuntu 16.04, the journal was Hot Garbage - to keep a production system working (as in, "didn't randomly stop logging, didn't regularly corrupt logs, didn't uncontrollably fill the disk because none of the limit options actually worked") we eventually backported about 30 fixes from newer versions - by 22.04 or so it was "fit for purpose" out of the box, but earlier than that it earned every bit of hatred it got.
    • guilhas 4mo ago
• NikhilVerma 4mo ago
  I have a Canon printer, I actually can't trust that their print nozzle won't get jammed up after sitting idle for a while. So I had claude setup a systemd script to print a picture of my dog every week, I ensure it has enough CMYK spectrum to stress the printer. Its a nice surprise every monday as I sit on my desk to see a sudden picture pop up from the printer :)
  • tdeck 4mo ago
    I wish printers could have a mode like this to print random images from an album, or a calendar, rather than wastefully draining ink into a sponge every few days. If nothing else, maybe it could be some kid's high school science fair project idea.
    • sunrunner 4mo ago
      How about printing a QR code for a randomly generated private key for Satoshi Nakamoto's Bitcoin wallet, then every few days you get a tiny moment of excitement, hope, and then disappointment. It's still wasteful, but it could pay off big time?
      • NSUserDefaults 4mo ago
        Or if you have a printer/scanner combo, you can turn it into a pen pal!
      • tekno45 4mo ago
        this is an amazing youtube video idea if you could get a type writer to do it.
        • sunrunner 4mo ago
          Maybe I'm misremembering, but I'm sure there was something on HN a few weeks ago about an electric typewriter that someone had connected to (I'm guessing) a Raspberry Pi? My search-fu is currently failing to find anything particularly recently, at the moment.
• frays 4mo ago
  I've been using Linux for over 20 years, systemd for over 10. Yet there's always something new to learn and actually consider as another useful tool.
  • egorfine 4mo ago
    I'm using Linux for about 30 years and apparently we were all wrong using cron for decades.
    • paulddraper 4mo ago
      Not wrong, but there’s room for improvement!
• sammy2255 4mo ago
  I've converted all my crons to systemd timers+services over the past year but cant help but think it's sort of.. less tangible than cron Like imagine trying to explain systemd timers and services and unit files to a beginner.
  • darkwater 4mo ago
    > Like imagine trying to explain systemd timers and services and unit files to a beginner. I think it's... easier? Like "systemd is the place where your system manages all the processes it needs to run. Part of those processes can be run on a schedule, or on a timer, and you define them using this simple text file".
  • PunchyHamster 4mo ago
    cron is easier for easy stuff ("just run this every 10 minutes") but harder for hard stuff ("run it every 8 hours but with randomized offset so not all machines at once do it, but also if machine was down when it should run, run it immediately"). It is also easier to debug as every job gets its own log rather than trying to write to system mailer nobody had set up with the job errors
• thomashabets2 4mo ago
  I haven't used systemd timers enough to disagree, but > Ambiguous $PATH settings make cron script execution difficult to predict. What makes you say that? You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else? > You might feel cool knowing the scheduling grammar by heart I've used Linux since 1994 and I don't know it by heart. But luckily it's pre-printed in the crontab as comments: # For more information see the manual pages of crontab(5) and cron(8) # # m h dom mon dow command You just put numbers aligned with the titles. The rest of the complaints, sure. Next time I need a cronjob, I'll try it out.
  • jchw 4mo ago
    The main nice thing about the environment in systemd is that it is standard and mostly a blank slate, whereas at least for me I was always getting bit by the fact that the environment in Crontab was completely different from say, the environment inherited by supervisord or sysvinit scripts. In systemd the actual unit that gets executed is the same regardless of what triggers it, so there is no gap. That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell. I'd have to concur that I agree this is an advantage of systemd.
    • skydhash 4mo ago
      I use cron in OpenBSD and it's a deterministic environment and mostly clean[0]. I like that instead of having other subsystems creep in. [0]: https://man.openbsd.org/crontab.5#ENVIRONMENT https://man.openbsd.org/crontab.5#ENVIRONMENT
    • simoncion 4mo ago
      > That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell. Odd. This script #!/bin/bash set > /tmp/set.txt when scheduled like so * * * * * $HOME/bin/testCronScript.sh Produces this file in /tmp/set.txt which has had a handful of values (HOME, UID, etc) lightly redacted prior to posting here -to remove PII or for length- but its keys are entirely untouched: BASH=/bin/bash BASHOPTS=<redacted because long> BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_LINENO=([0]="0") BASH_LOADABLES_PATH=/usr/local/lib64/bash:/usr/lib64/bash BASH_SOURCE=([0]="/home/user/bin/testCronScript.sh") BASH_VERSINFO=<redacted bash 5.3.x> BASH_VERSION=<redacted bash 5.3.x> DIRSTACK=() EUID=13370 GROUPS=() HOME=/home/user HOSTNAME=hostname HOSTTYPE=x86_64 IFS=$' \t\n' LANG=en_US.utf8 LOGNAME=user MACHTYPE=x86_64-pc-linux-gnu OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/bin:/bin:/usr/sbin:/sbin PPID=1337 PS4='+ ' PWD=/home/user SHELL=/bin/sh SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 TERM=dumb UID=13370 USER=user _=/home/user/bin/testCronScript.sh Seems pretty clean to me. Even when I run this via /etc/crontab, rather than as a user cron job: * * * * * root /home/user/bin/testCronScript.sh I get effectively the same results. Maybe your distro's default cron environment was bad, and you never bothered to check and unset the badness? I'd be surprised if they were unable to make the default environment for Timer Units to be bad.
• cmsj 4mo ago
  Is there a way yet to force-trigger a timer? There wasn't the last time I used them, which I found to be super annoying for testing them.
  • Biganon 4mo ago
    It's covered in the article, you can simply start the unit that would be started by the timer. Oh but it won't appear in the timer-specific logs, I guess...
• baggy_trough 4mo ago
  In decades of trying, I do not believe there was one time that I ever got a cron job to work properly in the first attempt. Systemd timers are a godsend.
• kayson 4mo ago
  I love systemd timers! I've slowly moved all of my ansible-deployed cron jobs to timers (now just an ansible copy!). The integration with journalctl, especially in a newer OS like Debian 13 where syslog is gone, is really nice. It's also really nice to be able to start the service manually for debug. Having a cron job that didn't work was an annoying exercise in copy/pasting or writing an extra shell script. Don't even get me started on the black hole of cron job stdout. I can monitor systemd services like I already do and get a notification on failure. I've noticed more and more open source projects recommending timers as a deployment method and I think that's great!
  • egorfine 4mo ago
    > more and more open source projects recommending timers I am perfectly happy with projects recommending timers as long as I can ignore them and use cron.
  • keynha 4mo ago
    [dead]
• sunshine-o 4mo ago
  This is actually something that I like in systemd. I am dealing with mostly non systemd system: BSD, Alpine, termux On BSD anacron works well, but I do not why I am always running into problems with the cronie anacron implementation. And it is very hard to debug. I would really like a simple modern cron/anacron alternative. Cronicle looked cool but it is node.js, a bit heavy and being replace now by their new product called xyOps anyway.
• mkesper 4mo ago
  There's another big feature: You're not relying on the time zone to which the server was set (like with cron) but can explicitly specify a time zone: https://www.freedesktop.org/software/systemd/man/latest/systemd.time.html# https://www.freedesktop.org/software/systemd/man/latest/syst...
  • 6031769 4mo ago
    Of course you can do this trivially in cron as well. It is what the CRON_TZ variable is for.
    • mkesper 4mo ago
      This is a GNU extension so not portable.
      • simoncion 4mo ago
        > This is a GNU extension so not portable. 1) It's supported by cronie. I bet it's supported by many other crons. 2) "Great" news! The software in the Systemd Project only officially runs on Linux, so "it's not portable" is a really bad counterargument when "alternatives to some Systemd Project feature" is the discussion topic.
  • eichin 4mo ago
    hmm, when did that get added? Last time I checked, the only timezone you could specify was UTC (which was one more than cron supported, but still insufficient.)
• mindcrime 4mo ago
  I haven't always been the biggest fan of systemd in some regards, but I will say that I mostly agree with this sentiment. I've almost completely quit using cron, and now favor systemd timers for scheduled jobs - at the "system" level anyway. I might still embed Quartz for scheduling that's scoped to a particular application or something. Why? It's one of those fuzzy and somewhat hard to explain things. The systemd approach just maps more cleanly to my mental model of "how things should work" I guess. And maybe some of it is that I did indeed experience plenty of " Ambiguous $PATH settings make cron script execution difficult to predict" in the past, although it's not just that. I won't sit here and claim that systemd timers are necessarily better than cron in any universal / objective sense. But they've won me over, for what it's worth.
• deleted 4mo ago
  [deleted]
• chuckadams 4mo ago
  I believe one of the major distro lines (redhat or debian, I forget which) uses systemd-cron, where cron is just a thin wrapper around systemd. You get more power from writing the unit files directly, but if all you ever need is a simple cron job, you have the old interface still available.
  • MathMonkeyMan 4mo ago
    Yep, I use this for a @reboot job and a few regular jobs on my home server. I use user crontabs, so I can get around the "unknown shell/path/etc." by prefixing every job with /some/shell -l myjob.sh or sometimes . ~/.profile && cd /some/where && ./job >>cron.log 2>&1
• pull_my_finger 4mo ago
  I'm fully ready to drink the "just let systemd do all the things" kool-aid, but I would love to see some sort of introductory/tutorial info into some of the things it can do other than services - i.e. containers and timers. I know man pages exist, but it would be nice if there was more scannable intro out there.
  • htx80nerd 4mo ago
    btw that xz hack only effect systemd distros
• dmuth 4mo ago
  > stdout and stderr output often ends up in a black hole Ain't that the truth. Literally every crontab I've written for the last 10 years has had this in it: 2>&1 | logger -t cron-WHATEVER ...and that does a pretty good job of capturing anything that the script emits and making it easy to grep for in syslog the following morning. But I'm still amazed at how many crontabs I run across that don't capture any output at all.
  • ninkendo 4mo ago
    The “standard” is for the output to go to your user’s mail box. You know, that thing you check with the “mail” command and has a user interface shockingly similar to `ed`. You check that all the time, right? Right? It’s… certainly a product of its time. (I have my system mailer set up to actually send mail to my Gmail account, with authenticated SMTP via API keys, which I did 15 years ago and have no recollection of how I even did it. It still works… somehow. I don’t even use Gmail any more, and I’ll be damned if I have to figure out how to do it with fastmail, and lord knows doing unauthenticated old-school SMTP is just gonna get sent to fastmail’s black hole, so that idea ain’t gonna work either.)
• 7e 4mo ago
  systemd is great all around. Don’t listen to the boomers complaining about it because their cheese was loved.
  • jjtheblunt 4mo ago
    (typo, i think). cheese was moved?
  • guilhas 4mo ago
    They should listen to other people talking about their knowledge about cheese, of the single one they know, the cheese slices from the supermarket
• progforlyfe 4mo ago
  This is a very good intro to systemd timers -- I think you convinced me to finally start using them. Love the "list-timers" thing as well. With cron, it never seemed easy to me to get a picture of all the cron jobs running on a box. I'd need to check crontab for all users, as well as /etc/cron.d/, as well as the daily/hourly/monthly directories. And in fact I do have a use-case for needing to run something ~5 minutes after the system boots and then every ~12 hours onward from there. It's great that systemd timers has me covered!
  • tylerjl 4mo ago
    Thanks for the kind words! Especially given how ubiquitous systemd is now, skilling up on the toolbox with commands like `systemd-analyze` and `systemctl list-timers` feels super valuable.
• simoncion 4mo ago
  You will love SystemD [0] timers until they fuck you over in an entirely inscrutable way and the SystemD maintainers don't care to either fix the problem or update the docs to warn of the shortcoming. One of our customers called in with a production down incident caused by a full disk. We got a copy of the VM and took a look. Investigation revealed that / was full because /var/log was full and that our 'logrotate' timer unit that was scheduled to run once a day had run either exactly never or exactly once... I can't remember which. Further investigation revealed no difference in software load or configuration between this VM and a VM that had a functional logrotate timer unit. Exactly one VM out of hundreds of identical VMs at this site (and many multiples of that at other customer's sites) were affected by this. Advising the customer to clear out /var/log and reboot did not unstick 'logrotate', and none of the diagnostics or fixes we could find anywhere unstuck it. Once "systemd-crond" decided to never schedule this job ever again, it stuck to that decision. After a lot of searching, we found an open bug report from a year or three prior where someone reported exactly the same symptoms and was scheduling a unit with pretty much the same set of unit configuration flags that we were using. The conversation from the core devs ran through the pattern that one gets used to seeing when one runs into SystemD bugs that are caused by extremely complex unanticipated interactions between parts of the project: "That's not a bug, only an idiot would want that to work.", "Oh, we don't document that that's not supposed to work?", "Wow, okay, yeah, I can see how that maybe should work. That it doesn't sure does seem weird.", "Having said that, I don't know if it's supposed to work, or if it's unsupported. Someone should really either document that or fix it."... and then the behavior is neither fixed nor documented. [1] Absent any actual explanation for the failure, we ended up swizzling the options in our 'logrotate' unit and praying that satisfied whatever gremlin arose from the depths to trouble our customer. SystemD contains an enormous -and ever-growing- amount of accidental complexity, and has a set of core maintainers who are generally disinterested in either documenting the places where one or more complex systems bind together to cause stop-the-world problems or fixing the systems involved so that they don't bind up. It's a fine project until it's very, very suddenly not, and then you're absolutely SOL. If you're lucky, you can shuffle around what you're doing [2] and hope that avoids the problem. [3] [0] Some folks use the spelling "SystemD" to mock the project. I use the spelling "SystemD" to distinguish between "the entire systemd project" and systemd(1). I do this because some folks will make a claim like "systemd is very, very small and self-contained. I don't understand why anyone would say otherwise.", but what they are actually saying is that systemd(1) is a fairly small program that doesn't do all that much when run as PID 1. It sucks minor amounts of ass that the project and the program it runs as PID 1 share the same name, but what can you do? [1] No, I don't have a link to the open bug report. This was more than a year ago, so the bug ID has been long forgotten. [2] The term of art for this practice is "wave a dead chicken at it". [3] Plus, like, even disregarding most of the rest of my report... how in the hell do you design a cron that knows a job is scheduled to be run periodically, can tell you how long it has been since it last ran, but never manages to run it? To me, that's unforgivable. It's a "You had one job!"-tier cockup.
• egorfine 4mo ago
  We have used cron perfectly fine for decades and it served us well within its very clear limitations. But now obviously we were so blind and wrong all this time and the only true solution is of course systemd.
  • bigbuppo 4mo ago
    Thank Lennart you degenerate apostates are finally starting to see the light of His glorious creation. Hallowed be thy systemd-journald.
  • bakugo 4mo ago
    Has it actually served you well? Because it hasn't served me well at all. I am not the biggest fan of systemd, but today I will always reach for a systemd timer over cron simply due to the sheer amount of bad experiences I've had with cron. Hours upon hours wasted trying to troubleshoot crons that weren't working due to some stupid obscure issue, having to use dirty hacks to monitor for success or retry failed jobs. A few years ago I was trying to run a very simple bash script with cron and the script just died halfway through for no reason. Nothing in logs, worked fine when run directly, but in cron it just stopped halfway through a loop. Never figured out the cause, just gave up and used a timer instead, which worked fine. Never touched cron again after that. The ease and convenience of monitoring and troubleshooting alone are worth switching over.
    • egorfine 4mo ago
      Let me state once again: "within its very clear limitations". Once you learn that env in cron is not same as in your shell and once you learn to redirect output to loggers - it works just fine. It would be a lie to say that I never debugged cron and sure it's annoying. > and the script just died halfway through for no reason Unrelated to cron. Bad script.
      • dijit 4mo ago
        I'm sympathetic, but "bad script" is an awful assertion. We are all guilty of making bad scripts, bash is a disgusting degenerate language (and I love it). The way we learn to write good scripts is by writing bad scripts in enough amounts to get bitten by all the warts. One thing I really love about cron, is that if you set up mail on the server (which: you should btw), then cron actually sends emails if it sees anything in stdout and stderr. I am a dyed in the wool systemd non-believer, but I really do like the timers.
• bigbuppo 4mo ago
  That would assume I like systemd in the first place.
• MarkusWandel 4mo ago
  Does systemd ship with something to upgrade your cron jobs for you? That would be the friendly way. Write your old school cron jobs, and then a script that converts them to do things the systemd way, documenting its steps, i.e. I created this file and this is why. Friendly "I help you do things better" rather than standoffish "your way is obsolete, you need to do it our way". Oh wait. I get it. LLM agents can do exactly that for you can't they. Another way I'm behind the curve. I have knocked together a systemd service or three based on google copypasta. But generally, for cron jobs, why make it complicated? One line in /etc/crontab and done. I generally call an encapsulation script that sets the right environment variables, uses absolute paths, captures stdout/stderr if required and so on. I just want the simplest possible way to launch that script on a schedule.
• Bender 4mo ago
  I will use what I am comfortable with and so should others. CronD, SystemD, atD, multiple conditional checks in a shell script, whatever tickles your fancy. There is no wrong answer, just document what you did and add a comment. Comments are permitted in cron. If someone keeps putting complex obfuscated time structures into cron make them decipher their incantation and keep nagging them until they keep it simple, comment their cron entries or until they and their manager resign. For what it's worth there are usually web apps popping up that can decipher goofy cron time/date incantations. [1] This one has a git repo in the top right, not my repo. Maybe clone it just in case their site goes away some day. [1] - https://crontab.cronhub.io/ https://crontab.cronhub.io/
• pjot 4mo ago
  After years of using orchestration tools like airflow and dagster so many lightbulbs have just lit up in my head. I wish documentation for tools would explain their abstractions concepts in terms of its primitives. Great post, thanks!
• jpcfl 4mo ago
  > Prime Time for a Timer Primer It's pronounced, "primmer."
  • OJFord 4mo ago
    TIL that's the standard US pronunciation. I thought you must be joking, referencing something else: OP's title works with BrE pronunciation at least. https://en.wiktionary.org/wiki/primer#Pronunciation https://en.wiktionary.org/wiki/primer#Pronunciation
    • KAMSPioneer 4mo ago
      Wait, really? I'm a native Midwestern/Great Plains American English speaker (I remember reading the Harry Potter books as a kid and wondering why all the -er words were spelled wrong) and I say "PRY-mer." I have never heard anyone say "PRIM-mer" in my life. Am...am I being punk'd...?
      • tmp10423288442 4mo ago
        I am too. I've heard of the supposedly correct pronunciation, but I can't bring myself to use it. The "PRY-mer" pronunciation is more common in practice.
      • chickensong 4mo ago
        I too have never once heard primmer. Not that all words follow the rules, but it doesn't make sense (vowel-consonant-vowel has long sounding vowel. That page also has the audio file links backwards (regular vs irregular). The file labeled irregular pronunciation sounds like primmer.
• louiskottmann 4mo ago
  And you immediately lose the ability to do `crontab -l` on any server to know its scheduled tasks. Now you get to look around the myriad of places where you can put systemd files, and figure out which ones are base services and which ones are custom, with no general convention to go about it. Nope.
  • arter45 4mo ago
    systemd list-timers With —-all
  • TazeTSchnitzel 4mo ago
    If you had read the article, you would have seen its answer to this.
  • Eduard 4mo ago
    /etc/crontab /etc/cron.d/* /etc/cron.hourly/* /etc/cron.daily/* /etc/cron.weekly/* /etc/cron.monthly/* /var/spool/cron/crontabs/*
    • zbentley 4mo ago
      Bad memories. I particularly enjoyed fighting with third-party programs that installed system cronjobs in the various tabs, and having to remember to go and find them after package upgrades and try to figure out how to robustly identify when their processes were running so my other cronjobs wouldn't overload or clobber state, since the third-party-installed jobs didn't play along with any lockfile-based coordination we used. Wants/WantedBy/Requires are godsends by comparison.
• encoderer 4mo ago
  I setup a few systemd timers last year and created https://systemd.guru https://systemd.guru to play with different options in OnCalendar expressions.
• t43562 4mo ago
  It may be a disastrous comment to make but I think I like cron better! A tool designed for a particular job etc.... :/
  • sophacles 4mo ago
    I designed a tool for flying. It's only designed for flying. It is based on the principles of the brick.
    • yoYo2R 4mo ago
      Cron. Initial release May 1975; 51 years ago It is succesfully flying 51 year. And will work next 50 years. Systemd probably will changes syntax in next 2 years. Modern development mindset: if tool is not rewritten last month - it is outdated and we need to reinvent it. Probably using blockchanin and AI.
      • t43562 4mo ago
        I don't love cron's time format - it's easy to make mistakes - but one-line, one file configuration is simple in a nice way. I bet we could make a cron that was easier but still simple.
        • yoYo2R 4mo ago
          Crontab could be confusing at first glance but 100 lines (literally) of man page have everything including examples. In my exerience they covers 99% of use cases unless you need something REALLY fancy. And you have cron on every system including BSD and not sure but probably also AIX/Sun/etc. It is universal and everyone knows about it. If your server doing something weird your probably will check crontab, not some obscure systemd subsystem almost no one knows about. Instead using standart solutions (POSIX by the way) systemd again found NIH problem with cron and added one more tool in init combain.
      • sophacles 4mo ago
        My point was purpose built doesn't mean its the best tool for the job. As for the rest of the drivel: so what it was used for a long time. That just means it was pretty good. That doesn't mean it has to stay forever, just that a new contender should do things better. Systemd timers address real shortcomings of cron. Your argument boils down to: everyone should be stuck with shortcomings of the early days of computing because you don't like new things.
• supriyo-biswas 4mo ago
  I wonder what happened with the heading, it was okay before, and then was mutilated since.
• porridgeraisin 4mo ago
  I am not the greatest fan of most of systemd's features. I will always prefer it tho since I just view it as a "packaging format". The same way I view docker. It is just that it happens to be the format that a lot of software is using and I have almost no headache integrating services, timers, logging and such of software I install. Without systemd its a mighty pain. Everyone uses the same one thing and that makes me overlook any drawbacks of the model. Only if the entire system was set up by me and mostly ran my software and I was getting paid for it, I might not use systemd. But one feature of systemd I will absolutely stand by is nspawn. It's just beautiful.
  • miladyincontrol 4mo ago
    Yeah nspawn has to be one the most underrated (and 100% optional ofc) components.
• KaiShips 4mo ago
  [flagged]
• BrenBarn 4mo ago
  I like these systemd things but I always find it annoying how I have to create multiple separate files (like a file for the service and another for the timer, or similar if I need a socket file). In theory this is more flexible but in practice it's vanishingly rare that I need the same service to be accessible to more than one timer. It would be nice if there were some alternative compound format that could combine the timer and service into one.
• tmp10423288442 4mo ago
  I command you to love systemd timers! (Yeah, they're pretty useful, especially after you get an LLM to write all the boilerplate for you. The boilerplate was the main reason I preferred crontab before.)
• arikrahman 4mo ago
  Refreshing to see a positive opinion with regards to systemd. Incidentally, my favorite way to spawn jobs, is well, the job spawn command in nushell.
• tylerjl 4mo ago
  Hey everyone, author here. I spotted the hn traffic a little late but I'm happy for any feedback or comments and will try and address the top-level comments as I can. (Aside: I wrote this article early last month but it caught on only just recently. For better or worse, touching a third rail topic like systemd seems like a sure-fire way to elicit strong and numerous reactions both positive and negative.)
• naikrovek 4mo ago
  Article author lost me when he demonstrated that two files are needed to schedule something. That is silly.
• opan 4mo ago
  >And yet. You probably shouldn't use literal cron (or its more modern cousins) for scheduled tasks! In 2026 there are more modern options available What are people on non-systemd distros like Guix System, Void, PCLinuxOS, and so on using for this? Is there still something better to use than cron? Admittedly I never learned cron, I use a lot of `sleep` and `countdown` for relative delays instead. Just earlier today I set up a 12h countdown followed by opening a URL with xdg-open since I expect a release around then and don't want to forget. I also threw in a little notify-send command in case my browser isn't visible, I should see that pop up. Considered using espeak, but don't wanna scare myself and/or ruin my watching experience if I'm watching a video at that time.
  • yoYo2R 4mo ago
    > What are people on non-systemd distros like Guix System, Void, PCLinuxOS, and so on using for this? Is there still something better to use than cron? Instead of `sleep` you can use `at`. But for scheduling `cron` is still the best. Package: at at, batch, atq, atrm - queue, examine, or delete jobs for later execution
  • egorfine 4mo ago
    > What are people on non-systemd distros like Guix System, Void, PCLinuxOS, and so on using for this? We have used and still use cron for decades. It does it's job and does it well.
• notorandit 4mo ago
  > In 2026 there are more modern options available More modern doesn't mean any better or more powerful. I for one hate this need for going "more modern" without having clear and factual advantages.
• Elhana 4mo ago
  Same shit, but you need to create 3 files instead of one line in crontab.