4 ms·

Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixe

Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixed time. With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available. Btw this is my repo for the backup automation: https://github.com/gchamon/borg-automated-backups https://github.com/gchamon/borg-automated-backups

mid-kid 5mo ago
Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules. To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this. Many cron implementations have a similar mechanism.
- gchamonlive 5mo ago
  EDITED This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
  - newsoftheday 5mo ago
    > fallback to running when the system comes online. That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described. I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then. > With @hourly I lose this control and multiple machines could potentially trigger backups at the same time Then don't use @hourly, use staggered times, it's very easy.
    - gchamonlive 5mo ago
      > That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described. This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.
      - happysadpanda2 5mo ago
        I interpreted it more like "I have these 500 different cronjobs all spread out across $unit_of_time. If the system is down for longer than $unit_of_time and then comes back, does all 500 jobs start running instantly (since they missed their previous deadline)?"
        gchamonlive 5mo ago
        Just to be clear, this isn't default systemd timer behaviour, you need to opt in by setting Persistent=true. If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool because at that scale you'd surely need better observability
        gchamonlive 4mo ago
        You could implement this with a gitlab instance in a separate system, like two VMs in proxmox or two physical machines, and a shell executor running in them. Gitlab CI has a nice feature to limit concurrency by using resource groups. Say you have 500 jobs spread through the day and the system stays offline for a while, when it comes online it'll start processing the jobs, but will only run a single one at a time. You get visibility, logs, queue monitoring and an API to query data.
        zbentley 4mo ago
        > If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool Eh sometimes, but you can get pretty far with one of two approaches: 1. Careful use of Requires= and Wants= to group your scripts into chains of jobs, which achieves fixed parallel (though at 100s of jobs, I hope you're generating those unit files with a tool like Puppet or https://github.com/karlicoss/dron https://github.com/karlicoss/dron or something and not doing this by hand). 2. Even better, just use a lockfile. `ExecStart="flock -F $TMPDIR/mylock <command>"` is pretty hard to beat. Use -F so as not to confuse KillMode and resource accounting and you're golden. Just don't use flock(1) timeouts; let systemd handle that. Heck, if you have that many cron jobs, you should be doing this even if you don't use systemd; otherwise job latency changes can cause reboot-style thundering herds out of the blue. If you need semaphore behavior and still don't want a real job queue, waitlock (https://github.com/bigattichouse/waitlock https://github.com/bigattichouse/waitlock) and many other CLIs have you covered.
  - layer8 5mo ago
    Cron also has @reboot. Not exactly the same, but has been sufficient for me so far.
    - gchamonlive 5mo ago
      https://news.ycombinator.com/item?id=48371021 https://news.ycombinator.com/item?id=48371021
newsoftheday 5mo ago
I'm sorry, I tried Googling the word "tolates" but I can't find any definition that makes sense? > runs the service as soons as the system is available. cron has the @reboot option which I use for a few scripts and works great.
- regularfry 5mo ago
  "tolerates".
- gchamonlive 5mo ago
  Typo, I meant tolerates. Fixed it. Not an option either, because if I reboot two machines and the backup starts in both of them it'll cripple my NAS
  - capitainenemo 4mo ago
    How does systemd on the 2 machines avoid that? Are they communicating somehow?
    - gchamonlive 4mo ago
      No, just different cron schedules. If I just reboot a machine the job doesn't get triggered, only if I start a machine after the cron schedule should have been triggered. To be fair, if I start two machines in these conditions this will happen too, but such situation is much more manageable than rebooting too machines in a short period of time.
michaelcampbell 5mo ago
> With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available. Cronie doesn't have a `@reboot` meta-trigger?
- gchamonlive 5mo ago
  https://news.ycombinator.com/item?id=48371021 https://news.ycombinator.com/item?id=48371021
WhyNotHugo 4mo ago
Snooze offers similar resilience to the system being offline: https://github.com/leahneukirchen/snooze https://github.com/leahneukirchen/snooze I find it very easy to reason about: a single process maps to a single recurring task. It can track last execution into a file, yielding durable schedules when the host is offline.

mid-kid 5mo ago

Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules. To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this. Many cron implementations have a similar mechanism.

gchamonlive 5mo ago
EDITED This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
- newsoftheday 5mo ago
  > fallback to running when the system comes online. That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described. I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then. > With @hourly I lose this control and multiple machines could potentially trigger backups at the same time Then don't use @hourly, use staggered times, it's very easy.
  - gchamonlive 5mo ago
    > That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described. This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.
    - happysadpanda2 5mo ago
      I interpreted it more like "I have these 500 different cronjobs all spread out across $unit_of_time. If the system is down for longer than $unit_of_time and then comes back, does all 500 jobs start running instantly (since they missed their previous deadline)?"
      - gchamonlive 5mo ago
        Just to be clear, this isn't default systemd timer behaviour, you need to opt in by setting Persistent=true. If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool because at that scale you'd surely need better observability
        gchamonlive 4mo ago
        You could implement this with a gitlab instance in a separate system, like two VMs in proxmox or two physical machines, and a shell executor running in them. Gitlab CI has a nice feature to limit concurrency by using resource groups. Say you have 500 jobs spread through the day and the system stays offline for a while, when it comes online it'll start processing the jobs, but will only run a single one at a time. You get visibility, logs, queue monitoring and an API to query data.
        zbentley 4mo ago
        > If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool Eh sometimes, but you can get pretty far with one of two approaches: 1. Careful use of Requires= and Wants= to group your scripts into chains of jobs, which achieves fixed parallel (though at 100s of jobs, I hope you're generating those unit files with a tool like Puppet or https://github.com/karlicoss/dron https://github.com/karlicoss/dron or something and not doing this by hand). 2. Even better, just use a lockfile. `ExecStart="flock -F $TMPDIR/mylock <command>"` is pretty hard to beat. Use -F so as not to confuse KillMode and resource accounting and you're golden. Just don't use flock(1) timeouts; let systemd handle that. Heck, if you have that many cron jobs, you should be doing this even if you don't use systemd; otherwise job latency changes can cause reboot-style thundering herds out of the blue. If you need semaphore behavior and still don't want a real job queue, waitlock (https://github.com/bigattichouse/waitlock https://github.com/bigattichouse/waitlock) and many other CLIs have you covered.
- layer8 5mo ago
  Cron also has @reboot. Not exactly the same, but has been sufficient for me so far.
  - gchamonlive 5mo ago
    https://news.ycombinator.com/item?id=48371021 https://news.ycombinator.com/item?id=48371021

newsoftheday 5mo ago

I'm sorry, I tried Googling the word "tolates" but I can't find any definition that makes sense? > runs the service as soons as the system is available. cron has the @reboot option which I use for a few scripts and works great.

regularfry 5mo ago
"tolerates".
gchamonlive 5mo ago
Typo, I meant tolerates. Fixed it. Not an option either, because if I reboot two machines and the backup starts in both of them it'll cripple my NAS
- capitainenemo 4mo ago
  How does systemd on the 2 machines avoid that? Are they communicating somehow?
  - gchamonlive 4mo ago
    No, just different cron schedules. If I just reboot a machine the job doesn't get triggered, only if I start a machine after the cron schedule should have been triggered. To be fair, if I start two machines in these conditions this will happen too, but such situation is much more manageable than rebooting too machines in a short period of time.

michaelcampbell 5mo ago

> With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available. Cronie doesn't have a `@reboot` meta-trigger?

gchamonlive 5mo ago
https://news.ycombinator.com/item?id=48371021 https://news.ycombinator.com/item?id=48371021

WhyNotHugo 4mo ago

Snooze offers similar resilience to the system being offline: https://github.com/leahneukirchen/snooze https://github.com/leahneukirchen/snooze I find it very easy to reason about: a single process maps to a single recurring task. It can track last execution into a file, yielding durable schedules when the host is offline.