4 ms·

cron running on only one server is unreliable and "high availability" cron is complex. So, it's unbelievable how no such solution existed before. It might not b

cron running on only one server is unreliable and "high availability" cron is complex. So, it's unbelievable how no such solution existed before. It might not be the most elegant way, but having a centralized cron (just like the database is centralized) helps a lot.

nickjj 5y ago
> So, it's unbelievable how no such solution existed before I think a lot of folks solved this problem at the application level using background workers or job processing services. For example, Celery with Python or Sidekiq with Ruby has had recurring job support for a really long time. Celery supported this since 2010. You could run distributed jobs (recurring or not) this way without using cron directly or requiring a custom DB extension. IMO it feels much more natural to solve this problem at the app level with a background worker since you can run any task you want, not just DB queries. This also has the added benefit of letting you use your DB library of choice (ORM, etc.) since you can run any code you want within the job.
- mey 5y ago
  In the Java world the venerable (pre-2006) Quartz library has been around a long time, and is certainly not the only solution. https://www.quartz-scheduler.org/overview/ https://www.quartz-scheduler.org/overview/
- luhn 5y ago
  Although Celery itself is distributed, the cron manager (celery-beat) is not and will not coordinate with other nodes. If you have two nodes running celery-beat, you'll get two jobs dispatched every time a cron triggers. imo defeats the whole purpose, at that point I might as well use crontab.
joshribakoff 5y ago
Perhaps you’re unaware, but kubernetes has cron jobs. Highly available, and a cron only runs on one node at a time (in the happy case)
- loevborg 5y ago
  > (in the happy case) That caveat makes a big difference. If you're making an API call like sending an email, you really don't want it to run twice.
  - williamdclt 5y ago
    Not sure it's the best example, having 0.01% of emails being sent twice is probably acceptable for most businesses. If it's a payment processing, it's not acceptable
    - giovannibonetti 5y ago
      > If it's a payment processing, it's not acceptable In the payment processing company I'm working at we deal with that independently. We set some data on Redis with a TTL to signal some request has already been made in the last X minutes, and if the same request arrives again during the same period the cached response is served
    - joshribakoff 5y ago
      hopefully your email/money movement service accepts an idempotency key which solves that
  - joexner 5y ago
    Idempotent resource cleanup tasks are the ideal use case for k8s crons.

nickjj 5y ago

> So, it's unbelievable how no such solution existed before I think a lot of folks solved this problem at the application level using background workers or job processing services. For example, Celery with Python or Sidekiq with Ruby has had recurring job support for a really long time. Celery supported this since 2010. You could run distributed jobs (recurring or not) this way without using cron directly or requiring a custom DB extension. IMO it feels much more natural to solve this problem at the app level with a background worker since you can run any task you want, not just DB queries. This also has the added benefit of letting you use your DB library of choice (ORM, etc.) since you can run any code you want within the job.

mey 5y ago
In the Java world the venerable (pre-2006) Quartz library has been around a long time, and is certainly not the only solution. https://www.quartz-scheduler.org/overview/ https://www.quartz-scheduler.org/overview/
luhn 5y ago
Although Celery itself is distributed, the cron manager (celery-beat) is not and will not coordinate with other nodes. If you have two nodes running celery-beat, you'll get two jobs dispatched every time a cron triggers. imo defeats the whole purpose, at that point I might as well use crontab.

joshribakoff 5y ago

Perhaps you’re unaware, but kubernetes has cron jobs. Highly available, and a cron only runs on one node at a time (in the happy case)

loevborg 5y ago
> (in the happy case) That caveat makes a big difference. If you're making an API call like sending an email, you really don't want it to run twice.
- williamdclt 5y ago
  Not sure it's the best example, having 0.01% of emails being sent twice is probably acceptable for most businesses. If it's a payment processing, it's not acceptable
  - giovannibonetti 5y ago
    > If it's a payment processing, it's not acceptable In the payment processing company I'm working at we deal with that independently. We set some data on Redis with a TTL to signal some request has already been made in the last X minutes, and if the same request arrives again during the same period the cached response is served
  - joshribakoff 5y ago
    hopefully your email/money movement service accepts an idempotency key which solves that
- joexner 5y ago
  Idempotent resource cleanup tasks are the ideal use case for k8s crons.