7 ms·
Prevent duplicate cron job running
- champtar 5y agoYou can also just use systemd timers :)
- lugged 5y agocron is a lot nicer imo. All your schedules are in one place and everyone knows how to access it.
- pabs3 5y agoThere is a systemd command for that: systemctl list-timers
- traceroute66 5y ago> cron is a lot nicer imo. How, exactly ? I've been a cron user forever, I had never heard of systemd timers until this thread. A quick internet search shows: - Nice clean config in a file (instead of having to edit crontab) - Lots of great features that you would have to manually implement and maintain in cron (e.g. missed launches, duplicate runs, random delay, timeouts) I mean sure, I've got bash script templates that let me solve the problem of preventing duplicate cron job running. But having seen how easy it is to achieve the same thing with systemd timers whilst not requiring a whole lot of boilerplate code in each shell file ... why wouldn't you switch to systemd timers ? I sure will be.
- jbverschoor 5y agoThe general consensus: "because it's systemd", "systemd is unstable", "systemd does too much", "systemd is non-unix". I like systemd. It's a good clone of launchd
- deleted 5y ago[deleted]
- binarybanana 5y agodcron is very lightweight and has most of those features. Dunno about random delay/timeouts, but worst case those sound like a 10 line shell script away. https://github.com/dubiousjim/dcron https://github.com/dubiousjim/dcron
- traceroute66 5y ago> dcron is very lightweight and has most of those features. Hmm, let's see ... - Choice 1: Use what comes with my system (two choices, systemd or, if for the anti-systemd people... cron) - Choice 2: Install some random third-party that has not had any substancial git commits since 2014 (the 2019 commits are just admin metadata).
- MertsA 5y agoYou can also just use systemd-cron :) https://github.com/systemd-cron/systemd-cron https://github.com/systemd-cron/systemd-cron
- ducktective 5y agoIs there a TUI to manage systemd-timers?
- sirius87 5y agoDid not know. Thanks for sharing this. This looks mighty useful.
- jbboehr 5y agoIf you use consul, you can use `consul lock` [0]. Allows you to have the cron configured on multiple servers too. [0]: https://www.consul.io/commands/lock https://www.consul.io/commands/lock
- darkwater 5y agoI don't get why people downvotes you. Clearly consul is not that widespread but if you happen to use it, consul lock is pretty cool, it also permits locks between a cluster of machines
- justinator 5y agoHuh, does python not have a way to flock a file itself? A simple flock doesn't always work all that well - the devil is in the details. And there is the issue that you miss a run of the script and have to wait another half hour for the next run. I've used the Highlander, "There can be only one!" code pattern for forever, and it works really well: https://www.perlmonks.org/?node_id=14260 https://www.perlmonks.org/?node_id=14260 You could change things to check on the status of your lock every minute or so for 15 minutes or whatever. Also in module form, https://metacpan.org/pod/App::Highlander https://metacpan.org/pod/App::Highlander And much more updated: https://medium.com/booking-com-development/highlander-b6717278a457 https://medium.com/booking-com-development/highlander-b67172... No need to attempt to stuff all this in a cronjob IMHO.
- knuthsat 5y agoPython has SQLite. This means you can lock a db file on all platforms and synchronize processes using SQLite.
- justinator 5y agoThat seems a little inefficient to pull in SQLite just so that you can have it lock a file, when you can just lock your own file, don't you think? glances around nervously is flocking one of those "what's old is new again" concept that people are just now relearning? Surely a blog post on how not to have two concurrent cronjobs running is kind of a weird thing to read when some Linux grey beard figured this out in the 90's on their lunch break.
- deleted 5y ago[deleted]
- knuthsat 5y agoSQLite comes for free in Python distribution. The only reason why I mentioned it. Of course, just having a nice cross-platform file locking interface would be nice (not the case for Python).
- pengaru 5y agoWelcome to Debian's dpkg::start-stop-daemon circa 2001 https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/utils/start-stop-daemon.c?id=d2c1c54171034f90ed4d3aba80b051f457fd569a https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/utils/start-s...
- 1500100900 5y agoOpenBSD's crontab provides a `-s command` flag which does similar thing in principle: https://man.openbsd.org/crontab.5 https://man.openbsd.org/crontab.5 "-s command Only a single instance of command will be run concurrently. Additional instances of command will not be scheduled until the earlier one completes." But I don't think it saves the state somewhere permanently, which isn't a big problem until cron's process dies for some reason.
- absc 5y agoIf you need to save state across executions, you should write a daemon instead.
- jbverschoor 5y agoThen "rm" should be a daemon too
- adrianmonk 5y agocron applies this very method of attempting to handle state by merely being a daemon, and it's not enough because cron can crash. The approach you're suggesting is what led to the problem you're proposing it as a solution for.
- pechay 5y agoAdd and verify a last_started and last_completed time to your local database.
- the2ndfloorguy 5y agoI have 100+ crons & I don't want to write some specific logic in my code for handling this, now what? would you mind sharing your thoughts on the same?
- pechay 5y agoThe other solutions based on file locking tools all seemed fragile. You have to change _something_!
- kbenson 5y agoOr install moreutils from your distro's package repo so you can use lckdo and get some other super useful shell helpers too. https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/
- eurasiantiger 5y agoOne would prefer not to lug a tool cabinet to a job site. Is there a standalone package for a tool like this?
- kbenson 5y agoWell, the flock command mentioned in the article will also work, but you might have to deal with having a bit more boilerplate around everything. The moreutils things are pretty small, some are just implemented as small perl scripts even. If you're writing the program, it's fairly easy to do yourself in it with Flock or Fcntl (you can usually just get an exclusive lock in the absolute path to the running executable/script. That said, quite a few of those moreutils items are useful in their own right (e.g. ts to prefix timestamps to each line of output is something so simple but so useful for logging output, chronic to silence stdout unless exit error is also very useful for cron, etc).
- eurasiantiger 5y agoBut doesn’t flock also require installing a utils package? The docker people, recently being forced to choose a life of mass migration, would greatly appreciate tools which do not introduce additional overhead to be carried across the whaling sea to the kubed podlands.
- kbenson 5y ago$ rpm -qi moreutils | grep Size Size : 194354 $ rpm -qi util-linux | grep Size Size : 11565734 So, < 200k compared to > 11MB based on compressed package, or if you want more specific counts for moreutils: $ rpm -ql moreutils | grep -v build-id | xargs du -hc 4.0K /usr/bin/chronic 4.0K /usr/bin/combine 20K /usr/bin/errno 24K /usr/bin/ifdata 16K /usr/bin/ifne 20K /usr/bin/isutf8 16K /usr/bin/lckdo 16K /usr/bin/mispipe 16K /usr/bin/pee 20K /usr/bin/sponge 8.0K /usr/bin/ts 8.0K /usr/bin/vidir 4.0K /usr/bin/vipe 4.0K /usr/bin/zrun 24K /usr/share/doc/moreutils 4.0K /usr/share/man/man1/chronic.1.gz ... 4.0K /usr/share/man/man1/zrun.1.gz 260K total The very small utils are actually perl scripts so there's a few dependencies it may pull in there (especially if there's no Perl in the container already, but this is for CentOS so people might be using different base containers. But honestly, a package made that excludes the perl portions would be ideal for containers. From what I can tell, there's no other libraries used except glibc, and then you get most of those implemented in ~20k or less each. $ rpm -ql moreutils | grep bin/ | xargs -n1 ldd | cut -d'(' -f1 | sort | uniq /lib64/ld-linux-x86-64.so.2 libc.so.6 => /lib64/libc.so.6 linux-vdso.so.1 not a dynamic executable If the perl was split into separate packages (CentOS already ships moreutils and moreutils-parallel to split that out), that would be ideal for containers for the portions that are binaries, IMO.
- aaronrobinson 5y agoSystemd is much more powerful than cron and manages your exact use case. You will need to deal with process hangs though if you have self dependencies like this.
- the2ndfloorguy 5y agoThanks for sharing. I will definitely look into this.
- rjh29 5y agoWe used to run crons using a custom wrapper which handled locking, logging, state reporting and enable/disable via puppet. However, systemd timer units basically do all of this for you, so I'd recommend that instead.
- the2ndfloorguy 5y agoCan you please share any resources to understand it more in detail? With flock, I am facing issue with lock files, if cron crashes or system reboots, lock files are a big pain.
- pritambaral 5y agoIME, flock works fine across SIGKILLs and forced reboots. Are you trying to use flock on a filesystem that runs across multiple kernels (i.e., networked fs), by any chance?
- tyingq 5y agoA fairly common pattern would be trying to create a specifically named file with the O_CREAT and O_EXCL flags. That's atomic, so you don't need flock. If it fails, then exit. This pattern is also often extended where you write your pid to the file and subsequent runs do a sanity check that your pid is actually running, and unlink the file if not. That said systemd-timers, as mentioned in many other comments, is probably the right answer here.
- nickcw 5y agoflock locks are released when the file is no longer open. This means that no matter how your command exits the OS will close open files and release the lock for you. The problem with creating files is that you then have the stale lock file problem to deal with which adds a lot more complication.
- the2ndfloorguy 5y agoIs there any other way without using lock files? Using socket or something?
- tyingq 5y agoI suppose you could try listening on a socket at a specific port. The next process would get EADDRINUSE if you were still running. There's also sysv or posix semaphores.
- klysm 5y agoI think systemd timers are vastly superior to cron mostly because of this problem.
- the2ndfloorguy 5y agoThank you. I will surely look into this.
- ckdarby 5y agoHow to use systemd timers as a replacement for cron jobs: https://opensource.com/article/20/7/systemd-timers https://opensource.com/article/20/7/systemd-timers
- bumpa 5y agoI have made a simple "cron with extras" project https://github.com/umputun/cronn https://github.com/umputun/cronn handling duplication issue as well as a bunch of other things, like restarting failed jobs, jitter, automatic reload and so on.