3 ms·

Cron is hard. I think it's better to just build in good locks and assume it will fail occasionally, and on failure repeat. This approach works on a small scale

Cron is hard. I think it's better to just build in good locks and assume it will fail occasionally, and on failure repeat. This approach works on a small scale, but it seems like the logic should be at the application layer. Secrets, API keys, ACL, etc all need to be duplicated on Lambda. Developers can't easily run it locally. Testing (eg integration testing on database migrations) is separate from the core app. Build promotion / rollbacks are different than the core application. Error tracking / logging may be different. Monitoring will be separate. Seems like a cool demo, but I think that running the code will have more overhead than writing it.

wheelerwj 10y ago
so, i'm not a very good engineer so maybe this is a stupid question... but i am going to ask it anyways. Cron is hard. Not in the sense that its challenging to set up but its a challenge to maintain and have visibility into. It seems like you need a whole set of infrastructure just to feel confident that things are running as intended. This has always lead me down the path of building a task manager type of micro-service running on an independent server to manage everything. But by the time im done, it feels like over kill. I just don't know what to do differently. Am I missing something?
- wmfiv 10y ago
  Something like http://www.easybatch.org/ http://www.easybatch.org/ may provide quite a bit of the infrastructure you're looking for. I'm sure there are equivalents for other languages.
- chrisp_dc 10y ago
  Sounds like you might be better off using Jenkins than roll your own. I use Jenkins over cron when I want easier auditing & authentication. I got the idea to use Jenkins as a 1-size fits all hammer from https://www.cloudbees.com/blog/drop-cron-use-jenkins-instead-scheduled-jobs https://www.cloudbees.com/blog/drop-cron-use-jenkins-instead...
  - philip1209 10y ago
    I've done this many times in the past. Add an authenticated "cron" endpoint, and have a jenkins job curl it every X seconds. It's not perfect, but it's an easy way to see log, alert Pagerduty, etc.
  - mickael-kerjean 10y ago
    Great to see I'm not the only one using Jenkins to look after crown jobs! However, it's to me way more than simply auditing & auth. The main advantage I found with Jenkins are: - you get a central place for all the cron jobs. That comes with all the advantages the most important one being nobody has to remember on which server is this freaking script - making an update can easily be done - running the task can even be done by non technical people. - easy backup - when a script become deprecated you can easily remove it and you're not very likely to let it there running for nothing forever - easy documentation for you tasks There's probably more but Jenkins is definitely awesome for cron jobs
  - flukus 10y ago
    Using a CI server as a scheduler? How is that at all the right tool for the job?
- philip1209 10y ago
  I've been burned by bad cron monitoring many times in the past. Short answer: check out https://cronitor.io/ https://cronitor.io/. Basically, finish every cron job by telling their server that the job finished. It's good enough monitoring for most cases.

wheelerwj 10y ago

so, i'm not a very good engineer so maybe this is a stupid question... but i am going to ask it anyways. Cron is hard. Not in the sense that its challenging to set up but its a challenge to maintain and have visibility into. It seems like you need a whole set of infrastructure just to feel confident that things are running as intended. This has always lead me down the path of building a task manager type of micro-service running on an independent server to manage everything. But by the time im done, it feels like over kill. I just don't know what to do differently. Am I missing something?

wmfiv 10y ago
Something like http://www.easybatch.org/ http://www.easybatch.org/ may provide quite a bit of the infrastructure you're looking for. I'm sure there are equivalents for other languages.
chrisp_dc 10y ago
Sounds like you might be better off using Jenkins than roll your own. I use Jenkins over cron when I want easier auditing & authentication. I got the idea to use Jenkins as a 1-size fits all hammer from https://www.cloudbees.com/blog/drop-cron-use-jenkins-instead-scheduled-jobs https://www.cloudbees.com/blog/drop-cron-use-jenkins-instead...
- philip1209 10y ago
  I've done this many times in the past. Add an authenticated "cron" endpoint, and have a jenkins job curl it every X seconds. It's not perfect, but it's an easy way to see log, alert Pagerduty, etc.
- mickael-kerjean 10y ago
  Great to see I'm not the only one using Jenkins to look after crown jobs! However, it's to me way more than simply auditing & auth. The main advantage I found with Jenkins are: - you get a central place for all the cron jobs. That comes with all the advantages the most important one being nobody has to remember on which server is this freaking script - making an update can easily be done - running the task can even be done by non technical people. - easy backup - when a script become deprecated you can easily remove it and you're not very likely to let it there running for nothing forever - easy documentation for you tasks There's probably more but Jenkins is definitely awesome for cron jobs
- flukus 10y ago
  Using a CI server as a scheduler? How is that at all the right tool for the job?
philip1209 10y ago
I've been burned by bad cron monitoring many times in the past. Short answer: check out https://cronitor.io/ https://cronitor.io/. Basically, finish every cron job by telling their server that the job finished. It's good enough monitoring for most cases.