4 ms·
> Here is an example of a typical oncall experience: 1) get paged with the command line of the failed job; 2) ssh into the scheduler machine; 3) copy & paste th
by foobarian 8y ago
> Here is an example of a typical oncall experience: 1) get paged with the command line of the failed job; 2) ssh into the scheduler machine; 3) copy & paste the command line to rerun the failed job
No doubt this works if it's an established procedure, but if I was approaching a system I wasn't familiar with I would never do (3) because the environment can differ wildly between crond and a login shell. It is safer to edit the cron schedule and duplicate the entry with a time set for few minutes in the future. (And clean that up afterwards).
- TheSoftwareGuy 8y agoYou could even write a cron job to clean up the crontab
- lykr0n 8y agoOr use Puppet/Salt/Chef/ansible to manage the file automatically
- toomuchtodo 8y agoI've seen this done previously. It gets arduous to maintain a large list of cron jobs, teach everyone how to add new ones, rebuild the cron box and redeploy on updates to config management. Just use Jenkins, until you outgrow Jenkins. Then use a distributed scheduler.
- rooam-dev 8y agoCan you recommend one? We recently had to implement a prototype in house, not that easy as I thought it would be, but we could not find a good solution, e.g. handle 100k jobs, long term (never expiring), short term (depends on an entity's live cycle), high available. Thanks.
- spiffytech 8y agoKubernetes does Cron jobs.
- toomuchtodo 8y agoKubernetes if you're a containers first shop, Hashicorp's Nomad if you're not just containers (can run binaries, etc).
- thwarted 8y agoOr you could use at(1) or batch(1) to invoke the job once. That seems significantly easier and less error prone than trying to clean up crontab.
- Pete_D 8y agoat(1) is great, but you'll still have the problem of a different environment to cron (it copies most of your environment variables and working directory).
- _kevinspencer 8y agoNot only that, but just ssh'ing in and blindly rerunning the failed job isn't the answer anyway. Research why it failed. If the job has to write a file on a full disk, you can rerun that thing a hundred times and it will never work. I'm sure they must have missed something from their write up as I can't imagine that's their on call playbook.