4 ms·

> Steveslist currently has a very simple and slightly fragile cron setup. We have a single “scheduled jobs server”. We use crontab on this server ... This setup

> Steveslist currently has a very simple and slightly fragile cron setup. We have a single “scheduled jobs server”. We use crontab on this server ... This setup isn’t scaling very well ... We’re considering setting up a new system using a modern tool like Kubernetes. Do people use Kubernetes for running scheduled jobs like this? It seems like it'd be overkill but in saying that I'm not sure if I know of anything that can be used for running scheduled tasks in a reliable and observable way that's scalable. Maybe Jenkins?

chucky_z 6y ago
this is a pretty common use-case for ci/cd tools. azure devops, jenkins, concourse, drone, gitlab, ... also a lot of etl tooling has this stuff (airflow comes to mind immediately) there's also nomad which is easy to run and can schedule a lot of jobs pretty quickly, with decent observability. there's... a surprising amount of tooling in this exact space!
harpratap 6y ago
The biggest Kubernetes cluster (arguably) is basically one huge batch job - https://cloud.google.com/blog/products/containers-kubernetes/google-kubernetes-engine-clusters-can-have-up-to-15000-nodes https://cloud.google.com/blog/products/containers-kubernetes...
cyberdrunk 6y ago
> Do people use Kubernetes for running scheduled jobs like this? We do, mostly because we already have Kubernetes and automation to deploy to it, so why not? The reliablity and monitoring are better than something we'd cook up on our own.
sciurus 6y ago
For better or worse, sure. Here's one case study- https://stripe.com/blog/operating-kubernetes https://stripe.com/blog/operating-kubernetes
seneca 6y ago
Kubernetes include cronjobs as an object type. Meaning, this functionality is built in. If you're already running Kubernetes, it's a very simple way to do schedule jobs without needing new management tech.

chucky_z 6y ago

this is a pretty common use-case for ci/cd tools. azure devops, jenkins, concourse, drone, gitlab, ... also a lot of etl tooling has this stuff (airflow comes to mind immediately) there's also nomad which is easy to run and can schedule a lot of jobs pretty quickly, with decent observability. there's... a surprising amount of tooling in this exact space!

harpratap 6y ago

The biggest Kubernetes cluster (arguably) is basically one huge batch job - https://cloud.google.com/blog/products/containers-kubernetes/google-kubernetes-engine-clusters-can-have-up-to-15000-nodes https://cloud.google.com/blog/products/containers-kubernetes...

cyberdrunk 6y ago

> Do people use Kubernetes for running scheduled jobs like this? We do, mostly because we already have Kubernetes and automation to deploy to it, so why not? The reliablity and monitoring are better than something we'd cook up on our own.

sciurus 6y ago

For better or worse, sure. Here's one case study- https://stripe.com/blog/operating-kubernetes https://stripe.com/blog/operating-kubernetes

seneca 6y ago

Kubernetes include cronjobs as an object type. Meaning, this functionality is built in. If you're already running Kubernetes, it's a very simple way to do schedule jobs without needing new management tech.