8 ms·

Cron best practices

155 points by cuu508 5y ago

ayushnix 5y ago
Can I implement something similar to systemd transient timers using cron jobs? https://wiki.archlinux.org/title/Systemd/Timers#Transient_timer_units https://wiki.archlinux.org/title/Systemd/Timers#Transient_ti... In addition, I also want to see the status of the transient cron job and see how much time is left before it executes, like `systemctl list-timers temporary.timer` would show.
- andylynch 5y ago
  I think that case is roughly covered by at(1)
_Anima_ 5y ago
At some point when the number of boxes and number of cronjobs become too high, I suggest looking at Rundeck as a central scheduler (not affiliated with PagerDuty, I actually work for a competiting company) or something similar. Also, you want to make sure your job has been running. I've used a selfhosted instance of healthchecks.io in order to warn me if something has not run.
recov 5y ago
When it gets big enough, airflow is easy to set up an much more powerful/useful
rr808 5y ago
Cron was great, but in the new world of cattle not pets, its obsolete.
- ckp95 5y ago
  What do you use instead?
  - rr808 5y ago
    we're on GCP so use Cloud Composer.
  - mmh0000 5y ago
    Cron is dead, long live cron[0] Fun fact, Kubernetes cronjobs are highly-available and fault-tolerant[1] [0] https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ https://kubernetes.io/docs/concepts/workloads/controllers/cr... [1] https://stripe.com/blog/operating-kubernetes https://stripe.com/blog/operating-kubernetes
macintux 5y ago
I fully expected one item to show up here, since this was shared at such an auspicious moment: don’t schedule any daily tasks between 1 and 2am local time, to avoid DST changes from either skipping or running them twice. (Unless cron has gotten smarter since last I used it, which was admittedly at least a decade ago.)
- blibble 5y ago
  most cron implementations have logic to handle it (whether or not you trust that is another thing...)
- chasil 5y ago
  Vixie cron (in the RedHat realm, written by Paul Vixie) addresses this. Local time changes of less than three hours, such as those caused by the Daylight Saving Time changes, are handled in a special way. This only applies to jobs that run at a specific time and jobs that run with a granularity greater than one hour. Jobs that run more frequently are scheduled normally. If time was adjusted one hour forward, those jobs that would have run in the interval that has been skipped will be run immediately. Con‐ versely, if time was adjusted backward, running the same job twice is avoided. Time changes of more than 3 hours are considered to be corrections to the clock or the timezone, and the new time is used immediately. It is possible to use different time zones for crontables. See crontab(5) for more information. Vixie crontabs also allow @reboot that enables any user with cron access to run programs at boot, which can be quite useful.
jrochkind1 5y ago
If only all tools were as good as curl at having flexible powerful options to control logging/error reporting/output.
- jrochkind1 5y ago
  To say more: I have sometimes had to resort to writing a wrapper script just for the purpose of gatekeeping what stdout/stderr a task in cron does (and ensure the exit code is properly (non-)zero. At one former job where I had a lot of cron jobs, I even wrote a re-usable wrapper utility for it. Many unix command line executables are pretty bad at it (especially if you deal with anything proprietary), so I get why people resort to just throwing away output from cronjobs, although it is a problematic lack of observability, true.
  - anotherevan 5y ago
    Would chronic solve this for you? https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/
rmoriz 5y ago
I love cronner to wrap and monitor cron jobs https://github.com/theckman/cronner https://github.com/theckman/cronner
BiteCode_dev 5y ago
They forgot the most important one IMO: document the intended duration and schedule of your cron in a comment above. Most people can't decipher the cron syntax by heart or without making a mistake, and among those most people, there is probably you when you are tired. Of course, there is always crontab guru (E.G: https://crontab.guru/#*/12_*_*_\ https://crontab.guru/#*/12_*_*_\*), but still, it's easier to find out with a comment next to it. Besides, it documents the intent, which may shows that there is a mismatch with the reality.
slotrans 5y ago
Don't use cron, use Jenkins.
withinboredom 5y ago
For as simple as the cron syntax is, the number of times I’ve seen a task scheduled to run every minute for an hour is far beyond non-zero.
- imtringued 5y ago
  It's because people want to test whether their command is actually successfully executed by cron because basic functionality like starting a job immediately isn't available and then they forget to revert that change. You have to use systemd timers for that... It's completely overkill (2 files each with a dozen lines) but it works much better in practice.
  - withinboredom 5y ago
    I don't think that's the problem, I just think its really easy to forget to set it to 0. I literally lol'd when I just happened to be reading the docs here just now: https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/ https://rancher.com/docs/k3s/latest/en/installation/install-... and saw it under the example cron for "--etcd-snapshot-schedule-cron."
deleted 5y ago
[deleted]
throwaway984393 5y ago
Locking is non-trivial, don't just use flock. Use the maildir method and check for stale locks; it's atomic, portable, and NFS-safe. Using Ohmycron is better than nothing, but it doesn't handle timeouts and stale locks and some other useful things. Seems like we still need a better universal wrapper for cron jobs.
- resizeitplz 5y ago
  > maildir method Guess I'm one of the lucky 10k today - but a search for information isn't turning up much. Could you expand on the maildir method a bit more?
  - 0xbadcafebee 5y ago
    https://manpages.ubuntu.com/manpages/xenial/man5/maildir.5.html https://manpages.ubuntu.com/manpages/xenial/man5/maildir.5.h... | https://manpages.ubuntu.com/manpages/xenial/man5/maildir.courier.5.html https://manpages.ubuntu.com/manpages/xenial/man5/maildir.cou... | https://gist.github.com/pwillis-els/b01b22f1b967a228c31db3cf2789ee13 https://gist.github.com/pwillis-els/b01b22f1b967a228c31db3cf...
xorcist 5y ago
There other variables in cron besides MAILTO, such as MAILFROM and SHELL. Checkout "man 5 crontab" for a concise listing. If cron runs on more than one host, it can be useful to add some randomness to the time to avoid the same job being run on exactly the same time everywhere. Red Hat-family distributions have RANDOM_DELAY variable for this. Or just add the delay when Ansible or a similar tool installs the job from version control. (Which you use, right?)
gjulianm 5y ago
My best practice with cron is to not use cron if possible. Systemd has timers with practically the same functionality and you'll never worry about: - Where to put the logs, they just go into the system journal. Easy to check, and if you already have alerts/monitors/whatever for the journal any new job integrates trivially. - Cron shell doing weird things. - Easy switching on/off of each timer. - Easy monitoring of when tasks are executed and the results. - Jobs don't get started if there's one already running (I think) - Better error messages when you mess the syntax. - More possibilities for scheduling. - Launching the job now if you need it to.
- traceroute66 5y ago
  > My best practice with cron is to not use cron if possible. Absolutely. I only randomly learnt about systemd timers thanks to a comment I read here on HN once. Since that point I've never looked back, its simply awesome. The main reason is its enabled me to remove a whole load of boilerplate from my scripts because systemd takes care of it for me. So my scripts are now shorter, more readable, and less prone to bugs. > Jobs don't get started if there's one already running Yes. This was the biggest selling point for me. On cron, I had a ton of boilerplate using all sorts of tricks to avoid the race condition of the script being run again before it had finished its prior execution. With systemd ? Remove the boilerplate, and, erm that's it. IIRC I don't think it even needs any special lines in your systemd unit file, it Just Works (tm).
  - merpkz 5y ago
    Well, to be fair this article touches exactly that and solution is - use flock command before your cronjob, hardly a ton of boilerplate.
    - traceroute66 5y ago
      > use flock command before your cronjob, hardly a ton of boilerplate Let's not go there shall we. Yes I know about flock, that's what I used. But if you are writing code correctly, you still end up with boilerplate. For a start, you need to write extra code to check the error status returned from flock. And then there's the overall point I was making. My "ton of boilerplate" referred to the overall boilerplate, i.e. flock was just one element, so then consider all the other boilerplate that was required to do all the other stuff that systemd does out of the box. Frankly, its a no-brainer. Systemd wins every time.

155 points by cuu508 5y ago

Can I implement something similar to systemd transient timers using cron jobs? https://wiki.archlinux.org/title/Systemd/Timers#Transient_timer_units https://wiki.archlinux.org/title/Systemd/Timers#Transient_ti... In addition, I also want to see the status of the transient cron job and see how much time is left before it executes, like `systemctl list-timers temporary.timer` would show.

andylynch 5y ago
I think that case is roughly covered by at(1)

_Anima_ 5y ago

At some point when the number of boxes and number of cronjobs become too high, I suggest looking at Rundeck as a central scheduler (not affiliated with PagerDuty, I actually work for a competiting company) or something similar. Also, you want to make sure your job has been running. I've used a selfhosted instance of healthchecks.io in order to warn me if something has not run.

recov 5y ago

When it gets big enough, airflow is easy to set up an much more powerful/useful

rr808 5y ago

Cron was great, but in the new world of cattle not pets, its obsolete.

ckp95 5y ago
What do you use instead?
- rr808 5y ago
  we're on GCP so use Cloud Composer.
- mmh0000 5y ago
  Cron is dead, long live cron[0] Fun fact, Kubernetes cronjobs are highly-available and fault-tolerant[1] [0] https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ https://kubernetes.io/docs/concepts/workloads/controllers/cr... [1] https://stripe.com/blog/operating-kubernetes https://stripe.com/blog/operating-kubernetes

macintux 5y ago

I fully expected one item to show up here, since this was shared at such an auspicious moment: don’t schedule any daily tasks between 1 and 2am local time, to avoid DST changes from either skipping or running them twice. (Unless cron has gotten smarter since last I used it, which was admittedly at least a decade ago.)

blibble 5y ago
most cron implementations have logic to handle it (whether or not you trust that is another thing...)
chasil 5y ago
Vixie cron (in the RedHat realm, written by Paul Vixie) addresses this. Local time changes of less than three hours, such as those caused by the Daylight Saving Time changes, are handled in a special way. This only applies to jobs that run at a specific time and jobs that run with a granularity greater than one hour. Jobs that run more frequently are scheduled normally. If time was adjusted one hour forward, those jobs that would have run in the interval that has been skipped will be run immediately. Con‐ versely, if time was adjusted backward, running the same job twice is avoided. Time changes of more than 3 hours are considered to be corrections to the clock or the timezone, and the new time is used immediately. It is possible to use different time zones for crontables. See crontab(5) for more information. Vixie crontabs also allow @reboot that enables any user with cron access to run programs at boot, which can be quite useful.

jrochkind1 5y ago

If only all tools were as good as curl at having flexible powerful options to control logging/error reporting/output.

jrochkind1 5y ago
To say more: I have sometimes had to resort to writing a wrapper script just for the purpose of gatekeeping what stdout/stderr a task in cron does (and ensure the exit code is properly (non-)zero. At one former job where I had a lot of cron jobs, I even wrote a re-usable wrapper utility for it. Many unix command line executables are pretty bad at it (especially if you deal with anything proprietary), so I get why people resort to just throwing away output from cronjobs, although it is a problematic lack of observability, true.
- anotherevan 5y ago
  Would chronic solve this for you? https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/

rmoriz 5y ago

I love cronner to wrap and monitor cron jobs https://github.com/theckman/cronner https://github.com/theckman/cronner

BiteCode_dev 5y ago

They forgot the most important one IMO: document the intended duration and schedule of your cron in a comment above. Most people can't decipher the cron syntax by heart or without making a mistake, and among those most people, there is probably you when you are tired. Of course, there is always crontab guru (E.G: https://crontab.guru/#*/12_*_*_\ https://crontab.guru/#*/12_*_*_\*), but still, it's easier to find out with a comment next to it. Besides, it documents the intent, which may shows that there is a mismatch with the reality.

slotrans 5y ago

Don't use cron, use Jenkins.

withinboredom 5y ago

For as simple as the cron syntax is, the number of times I’ve seen a task scheduled to run every minute for an hour is far beyond non-zero.

imtringued 5y ago
It's because people want to test whether their command is actually successfully executed by cron because basic functionality like starting a job immediately isn't available and then they forget to revert that change. You have to use systemd timers for that... It's completely overkill (2 files each with a dozen lines) but it works much better in practice.
- withinboredom 5y ago
  I don't think that's the problem, I just think its really easy to forget to set it to 0. I literally lol'd when I just happened to be reading the docs here just now: https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/ https://rancher.com/docs/k3s/latest/en/installation/install-... and saw it under the example cron for "--etcd-snapshot-schedule-cron."

deleted 5y ago

[deleted]

throwaway984393 5y ago

Locking is non-trivial, don't just use flock. Use the maildir method and check for stale locks; it's atomic, portable, and NFS-safe. Using Ohmycron is better than nothing, but it doesn't handle timeouts and stale locks and some other useful things. Seems like we still need a better universal wrapper for cron jobs.

resizeitplz 5y ago
> maildir method Guess I'm one of the lucky 10k today - but a search for information isn't turning up much. Could you expand on the maildir method a bit more?
- 0xbadcafebee 5y ago
  https://manpages.ubuntu.com/manpages/xenial/man5/maildir.5.html https://manpages.ubuntu.com/manpages/xenial/man5/maildir.5.h... | https://manpages.ubuntu.com/manpages/xenial/man5/maildir.courier.5.html https://manpages.ubuntu.com/manpages/xenial/man5/maildir.cou... | https://gist.github.com/pwillis-els/b01b22f1b967a228c31db3cf2789ee13 https://gist.github.com/pwillis-els/b01b22f1b967a228c31db3cf...

xorcist 5y ago

There other variables in cron besides MAILTO, such as MAILFROM and SHELL. Checkout "man 5 crontab" for a concise listing. If cron runs on more than one host, it can be useful to add some randomness to the time to avoid the same job being run on exactly the same time everywhere. Red Hat-family distributions have RANDOM_DELAY variable for this. Or just add the delay when Ansible or a similar tool installs the job from version control. (Which you use, right?)

gjulianm 5y ago

My best practice with cron is to not use cron if possible. Systemd has timers with practically the same functionality and you'll never worry about: - Where to put the logs, they just go into the system journal. Easy to check, and if you already have alerts/monitors/whatever for the journal any new job integrates trivially. - Cron shell doing weird things. - Easy switching on/off of each timer. - Easy monitoring of when tasks are executed and the results. - Jobs don't get started if there's one already running (I think) - Better error messages when you mess the syntax. - More possibilities for scheduling. - Launching the job now if you need it to.

traceroute66 5y ago
> My best practice with cron is to not use cron if possible. Absolutely. I only randomly learnt about systemd timers thanks to a comment I read here on HN once. Since that point I've never looked back, its simply awesome. The main reason is its enabled me to remove a whole load of boilerplate from my scripts because systemd takes care of it for me. So my scripts are now shorter, more readable, and less prone to bugs. > Jobs don't get started if there's one already running Yes. This was the biggest selling point for me. On cron, I had a ton of boilerplate using all sorts of tricks to avoid the race condition of the script being run again before it had finished its prior execution. With systemd ? Remove the boilerplate, and, erm that's it. IIRC I don't think it even needs any special lines in your systemd unit file, it Just Works (tm).
- merpkz 5y ago
  Well, to be fair this article touches exactly that and solution is - use flock command before your cronjob, hardly a ton of boilerplate.
  - traceroute66 5y ago
    > use flock command before your cronjob, hardly a ton of boilerplate Let's not go there shall we. Yes I know about flock, that's what I used. But if you are writing code correctly, you still end up with boilerplate. For a start, you need to write extra code to check the error status returned from flock. And then there's the overall point I was making. My "ton of boilerplate" referred to the overall boilerplate, i.e. flock was just one element, so then consider all the other boilerplate that was required to do all the other stuff that systemd does out of the box. Frankly, its a no-brainer. Systemd wins every time.