4 ms·

I would be really interested in a distributed job running system with a scheduler. I want a UI where I can set jobs via a schedule that run on one of many work

I would be really interested in a distributed job running system with a scheduler. I want a UI where I can set jobs via a schedule that run on one of many workers. In that same UI I want to be able to look at failed jobs and see all of the output from that job. I don't want to have to ssh over to the box where the job ran to retrieve a log file. I once had a job where we used Tidal Enterprise Scheduler which poorly implemented this. What solutions exist for this workflow in the open source world?

Barreljack 8y ago
Not quite distributed, but https://www.rundeck.com/open-source https://www.rundeck.com/open-source is pretty nifty.
yzmtf2008 8y ago
Look into airflow: https://airflow.apache.org/ https://airflow.apache.org/
TomMarius 8y ago
Would you pay for this? How much and what pricing scheme would you prefer?
- paddy_m 8y ago
  As a service that runs on Heroku I'd pay around $25/month. that'd be $25/month * 2 (stage+prod). At the bootstrapping stage i'm at now, that's what I'd pay. If I were profitable, I'd pay up to $500 - $1000 a month and happily so, figure out market segmentation so that I'd be forced to pay that much. I would probably run around 1000 jobs a month, which seems like a number that would put me in a higher pricing range. This service would be super useful for around 60 jobs a month even. Output from the jobs would be tiny, maybe 10k at most. I would be happy if output was saved on private S3 buckets. I guess the big point is, I'm looking to use this for regularly scheduled jobs, not just as a queuing service to spread load. My email is in my profile if you want to get in touch.
xorcist 8y ago
Rundeck does exactly this. If you already have something like Jenkins you could probably bend that to your will, too.
nkrumm 8y ago
Also ndscheduler: https://github.com/Nextdoor/ndscheduler/ https://github.com/Nextdoor/ndscheduler/
bloopernova 8y ago
Rundeck, as others have indicated, will do just that. I've used Tidal too, it's utterly horrible and universally hated by anyone I know who has used it. What took months to set up and debug in Tidal, took us hours in Rundeck.

Barreljack 8y ago

Not quite distributed, but https://www.rundeck.com/open-source https://www.rundeck.com/open-source is pretty nifty.

yzmtf2008 8y ago

Look into airflow: https://airflow.apache.org/ https://airflow.apache.org/

TomMarius 8y ago

Would you pay for this? How much and what pricing scheme would you prefer?

paddy_m 8y ago
As a service that runs on Heroku I'd pay around $25/month. that'd be $25/month * 2 (stage+prod). At the bootstrapping stage i'm at now, that's what I'd pay. If I were profitable, I'd pay up to $500 - $1000 a month and happily so, figure out market segmentation so that I'd be forced to pay that much. I would probably run around 1000 jobs a month, which seems like a number that would put me in a higher pricing range. This service would be super useful for around 60 jobs a month even. Output from the jobs would be tiny, maybe 10k at most. I would be happy if output was saved on private S3 buckets. I guess the big point is, I'm looking to use this for regularly scheduled jobs, not just as a queuing service to spread load. My email is in my profile if you want to get in touch.

xorcist 8y ago

Rundeck does exactly this. If you already have something like Jenkins you could probably bend that to your will, too.

nkrumm 8y ago

Also ndscheduler: https://github.com/Nextdoor/ndscheduler/ https://github.com/Nextdoor/ndscheduler/

bloopernova 8y ago

Rundeck, as others have indicated, will do just that. I've used Tidal too, it's utterly horrible and universally hated by anyone I know who has used it. What took months to set up and debug in Tidal, took us hours in Rundeck.