5 ms·

Ask HN: Job Scheduling as a service

23 points by mani-smhack 9y ago

Is there any product that will allow us to schedule via an api , and then if the time is triggered then it should call the destination via webhook(with payload preferably ) Right now we are using quartz scheduler which has more configuration (Including Mysql Connection ) . Cons are 1) Does not have an admin UI 2) No reporting or history of jobs Thanks

stephenr 9y ago
You want cron, as a service? Shit on a stick doesn't anyone actually host anything themselves anymore?
- dozzie 9y ago
  We do, but it's not sexy, because it just gets the job done and you can't really brag about not having bugs or troubles.
- mani-smhack 9y ago
  I need a scheduler where i can dynamically ( via api ) schedule a job , with the admin UI to manage the jobs . I Think these are not part of application development and it can be offloaded to hosted services .
  - caraga 9y ago
    Jenkins has scheduled jobs and an UI. It can also be accessed via API. https://jenkins.io/ https://jenkins.io/
    - mani-smhack 9y ago
      This is not with respect to devops or build . My app has schedule functionality for example a scheduled report , so if it where like a webhook call then i dont want to waste the cpu cycle ,now i am using quartz scheduler in my java app .
      - aprdm 9y ago
        Jenkins isn't just for devops or build.
  - bpicolo 9y ago
    There are a lot of frameworks for this in various languages, depends on what you're doing - like sidekiq in ruby and hangfire in dotnet https://github.com/mperham/sidekiq https://github.com/mperham/sidekiq https://www.hangfire.io/ https://www.hangfire.io/
- ddorian43 9y ago
  But cron isn't highly available/distributed with consistent state/scheduling
  - stephenr 9y ago
    Run cron on all your boxes, have them pull a job at a time from the database. Or shit, go full tilt with a clustered Redis job queue. All of these are better options than "embed the entire job info in a URL and hand it off to a remote scheduling service".
    - scarface74 9y ago
      http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html?m=1 http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant...
      - stephenr 9y ago
        The user doesn't need a queue though. They just need a process to find posts with a publish date of <= now that aren't published and publish them. That's a one line indempotent sql update statement in any decently architected application.
        scarface74 9y ago
        This was the other part of the comment I replied to: Run cron on all your boxes, have them pull a job at a time from the database. Or shit, go full tilt with a clustered Redis job queue. You're solving for the example but the original poster noted multiple jobs.
skibz 9y ago
Operated by my coworker: https://crontron.com https://crontron.com
stevekemp 9y ago
There are a few online tools, such as: https://www.easycron.com/ https://www.easycron.com/ https://www.setcronjob.com/ https://www.setcronjob.com/ etc. You might be as well writing your own though, since it shouldn't be a difficult task.
- scarface74 9y ago
  I couldn't agree more, I am a Dev lead with a mandate to "outsource everything possible" When it comes to infrastructure type stuff, but even he would look at me crazy for outsourcing job scheduling.
- egamirorrim 9y ago
  Even if you want clustered level resilience? Loads of places i've worked end up running a scheduler on 'just one server' - that ain't redundant brother!
cdoxsey 9y ago
Google app engine has a cron service: https://cloud.google.com/appengine/docs/standard/python/config/cronref https://cloud.google.com/appengine/docs/standard/python/conf... You'll have to create an app handler for the URL which can then make the outbound http request. It's inexpensive, reliable and has an admin interface and logs.
- mani-smhack 9y ago
  Yeah this works for cron jobs . Let me explain my requirement , say i have social media platform and users will schedule their post , and i need to schedule it in my scheduler and i have to persist in the databsae too . Because if the server is restarted all my scheduled jobs will be gone .. So i am looking for a hosted scheduler where i can send a rest API request with a webhook and a small payload . During the execution time the hosted scheduler will call the webhook with the payload , so that i will publish the post . Pros is that i dont want to worry about the scheduled jobs .Right now i am unable to find a scheduler with the good admin UI ( to pause all the jobs ) to delete the trial users post .
  - stephenr 9y ago
    This sounds like a terrible way to achieve your goal. Save the future posts to the database with the required publish date/time. Run cron every $X minutes where $X is the smallest interval you allow (ie if you force publish date to be rounded to 5 minutes, run cron every 5 minutes) and check for "future posts" where the publish date <= now(), and take action. This is pretty basic stuff and that you want to offload it to a 3rd party by capturing all the details into a request to be made back to your app is a huge red flag IMO.
    - scarface74 9y ago
      He's posting an example. He mentioned multiple jobs.
  - scarface74 9y ago
    Because if the server is restarted all my scheduled jobs will be gone .. I went into more details about Hashicorp's Nomad in another post, but I didn't mention that you can run a cluster of Nomad servers for high availability. Nomad is open source - I don't have a dog in the fight - I'm just a happy user.
podhy 9y ago
Rundeck
scarface74 9y ago
Nomad by Hashicorp. You have to do just a little legwork to get it to call an API - just schedule a job that calls the appropriate curl command. When I'm explaining it to people, I explain it is a "distributed cron". You schedule a job either using the command line or via an API to the Nomad server and then it runs the job on any box that has the Nomad agent running and registered to the server. If you combine Nomad with Consul, configuration is dead simple. https://www.nomadproject.io https://www.nomadproject.io
- bpicolo 9y ago
  Nomad is probably a bit overkill for cron. It's a whole k8s alternative isn't it?
  - scarface74 9y ago
    It can be. But if you just need a distributed cron, you can just use the raw_exec or exec driver type and run a shell command.
Daviey 9y ago
I agree that Nomad is a GREAT solution for this... but another thing to look at is Rundeck http://rundeck.org/ http://rundeck.org/
swindsor 9y ago
AWS’s Cloud Watch Events take Cron expressions and can trigger various actions (Lambda, ECS task, SQS/SNS message, etc). http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Wh... http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-e... http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/s...
aaronhoffman 9y ago
Something like Azure Storage Queues and Functions may be what you're looking for https://aaron-hoffman.blogspot.com/2017/01/replace-azure-scheduler-with-azure.html https://aaron-hoffman.blogspot.com/2017/01/replace-azure-sch...
BerislavLopac 9y ago
Apache Airflow? It's not SaaS but is easily self-hosted, and I'm aware of a few initiatives to make it into a service.
z0mbie42 9y ago
with a friend we did : https://www.jobscheduler.xyz/ https://www.jobscheduler.xyz/ it's an MVP but it works well for our projects (chatbot, emailing service...)
guilhas 9y ago
I would use Jenkins, although marketed as CI tool, can do what you asked. And has a lot of customization. https://jenkins.io/ https://jenkins.io/
aprdm 9y ago
You can use Jenkins.
macca321 9y ago
https://azure.microsoft.com/en-gb/services/scheduler/ https://azure.microsoft.com/en-gb/services/scheduler/
_Marak_ 9y ago
I think hook.io can do this. see: http://hook.io/cron http://hook.io/cron
mani-smhack 9y ago
Basically what i am looking is a quartz scheduler alternative with monitoring and admin UI(preferrably hosted) More like sendgrid.com for scheduling
sahilagg06 9y ago
https://github.com/flipkart-incubator/simpleJobScheduler https://github.com/flipkart-incubator/simpleJobScheduler
ako 9y ago
Redwood software RunMyJob: https://www.redwood.com/runmyjobs/ https://www.redwood.com/runmyjobs/
egamirorrim 9y ago
I'm surprised no one's mentioned Chronos - it takes a butt load of setup because it runs on top of Mesos (which requires Zookeeper) etc - but if you already have these tools in your stack it's a pretty tidy clustered scheduler: https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d https://medium.com/airbnb-engineering/chronos-a-replacement-... Edit: if you haven't got this stack around though i'd second calls to use a build tool like Jenkins or Rundeck - of course you probably don't want this on the same Jenkins that builds your jobs... so you'd have to set all that up
assafmo 9y ago
webhook.io works great for me. they also have a great free plan
- jnbiche 9y ago
  webhook.io is now defunct. They either shut it already down or will soon shut it down. I think webtask.io is somewhat similar. Also, iron.io lets you schedule functions using docker, has full API. AWS Lambda also lets you schedule Lambda and has full API.
  - assafmo 9y ago
    sorry i meant webtask.io!

23 points by mani-smhack 9y ago

stephenr 9y ago

You want cron, as a service? Shit on a stick doesn't anyone actually host anything themselves anymore?

dozzie 9y ago
We do, but it's not sexy, because it just gets the job done and you can't really brag about not having bugs or troubles.
mani-smhack 9y ago
I need a scheduler where i can dynamically ( via api ) schedule a job , with the admin UI to manage the jobs . I Think these are not part of application development and it can be offloaded to hosted services .
- caraga 9y ago
  Jenkins has scheduled jobs and an UI. It can also be accessed via API. https://jenkins.io/ https://jenkins.io/
  - mani-smhack 9y ago
    This is not with respect to devops or build . My app has schedule functionality for example a scheduled report , so if it where like a webhook call then i dont want to waste the cpu cycle ,now i am using quartz scheduler in my java app .
    - aprdm 9y ago
      Jenkins isn't just for devops or build.
- bpicolo 9y ago
  There are a lot of frameworks for this in various languages, depends on what you're doing - like sidekiq in ruby and hangfire in dotnet https://github.com/mperham/sidekiq https://github.com/mperham/sidekiq https://www.hangfire.io/ https://www.hangfire.io/
ddorian43 9y ago
But cron isn't highly available/distributed with consistent state/scheduling
- stephenr 9y ago
  Run cron on all your boxes, have them pull a job at a time from the database. Or shit, go full tilt with a clustered Redis job queue. All of these are better options than "embed the entire job info in a URL and hand it off to a remote scheduling service".
  - scarface74 9y ago
    http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html?m=1 http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant...
    - stephenr 9y ago
      The user doesn't need a queue though. They just need a process to find posts with a publish date of <= now that aren't published and publish them. That's a one line indempotent sql update statement in any decently architected application.
      - scarface74 9y ago
        This was the other part of the comment I replied to: Run cron on all your boxes, have them pull a job at a time from the database. Or shit, go full tilt with a clustered Redis job queue. You're solving for the example but the original poster noted multiple jobs.

skibz 9y ago

Operated by my coworker: https://crontron.com https://crontron.com

stevekemp 9y ago

There are a few online tools, such as: https://www.easycron.com/ https://www.easycron.com/ https://www.setcronjob.com/ https://www.setcronjob.com/ etc. You might be as well writing your own though, since it shouldn't be a difficult task.

scarface74 9y ago
I couldn't agree more, I am a Dev lead with a mandate to "outsource everything possible" When it comes to infrastructure type stuff, but even he would look at me crazy for outsourcing job scheduling.
egamirorrim 9y ago
Even if you want clustered level resilience? Loads of places i've worked end up running a scheduler on 'just one server' - that ain't redundant brother!

cdoxsey 9y ago

Google app engine has a cron service: https://cloud.google.com/appengine/docs/standard/python/config/cronref https://cloud.google.com/appengine/docs/standard/python/conf... You'll have to create an app handler for the URL which can then make the outbound http request. It's inexpensive, reliable and has an admin interface and logs.

mani-smhack 9y ago
Yeah this works for cron jobs . Let me explain my requirement , say i have social media platform and users will schedule their post , and i need to schedule it in my scheduler and i have to persist in the databsae too . Because if the server is restarted all my scheduled jobs will be gone .. So i am looking for a hosted scheduler where i can send a rest API request with a webhook and a small payload . During the execution time the hosted scheduler will call the webhook with the payload , so that i will publish the post . Pros is that i dont want to worry about the scheduled jobs .Right now i am unable to find a scheduler with the good admin UI ( to pause all the jobs ) to delete the trial users post .
- stephenr 9y ago
  This sounds like a terrible way to achieve your goal. Save the future posts to the database with the required publish date/time. Run cron every $X minutes where $X is the smallest interval you allow (ie if you force publish date to be rounded to 5 minutes, run cron every 5 minutes) and check for "future posts" where the publish date <= now(), and take action. This is pretty basic stuff and that you want to offload it to a 3rd party by capturing all the details into a request to be made back to your app is a huge red flag IMO.
  - scarface74 9y ago
    He's posting an example. He mentioned multiple jobs.
- scarface74 9y ago
  Because if the server is restarted all my scheduled jobs will be gone .. I went into more details about Hashicorp's Nomad in another post, but I didn't mention that you can run a cluster of Nomad servers for high availability. Nomad is open source - I don't have a dog in the fight - I'm just a happy user.

podhy 9y ago

Rundeck

scarface74 9y ago

Nomad by Hashicorp. You have to do just a little legwork to get it to call an API - just schedule a job that calls the appropriate curl command. When I'm explaining it to people, I explain it is a "distributed cron". You schedule a job either using the command line or via an API to the Nomad server and then it runs the job on any box that has the Nomad agent running and registered to the server. If you combine Nomad with Consul, configuration is dead simple. https://www.nomadproject.io https://www.nomadproject.io

bpicolo 9y ago
Nomad is probably a bit overkill for cron. It's a whole k8s alternative isn't it?
- scarface74 9y ago
  It can be. But if you just need a distributed cron, you can just use the raw_exec or exec driver type and run a shell command.

Daviey 9y ago

I agree that Nomad is a GREAT solution for this... but another thing to look at is Rundeck http://rundeck.org/ http://rundeck.org/

swindsor 9y ago

AWS’s Cloud Watch Events take Cron expressions and can trigger various actions (Lambda, ECS task, SQS/SNS message, etc). http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Wh... http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-e... http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/s...

aaronhoffman 9y ago

Something like Azure Storage Queues and Functions may be what you're looking for https://aaron-hoffman.blogspot.com/2017/01/replace-azure-scheduler-with-azure.html https://aaron-hoffman.blogspot.com/2017/01/replace-azure-sch...

BerislavLopac 9y ago

Apache Airflow? It's not SaaS but is easily self-hosted, and I'm aware of a few initiatives to make it into a service.

z0mbie42 9y ago

with a friend we did : https://www.jobscheduler.xyz/ https://www.jobscheduler.xyz/ it's an MVP but it works well for our projects (chatbot, emailing service...)

guilhas 9y ago

I would use Jenkins, although marketed as CI tool, can do what you asked. And has a lot of customization. https://jenkins.io/ https://jenkins.io/

aprdm 9y ago

You can use Jenkins.

macca321 9y ago

https://azure.microsoft.com/en-gb/services/scheduler/ https://azure.microsoft.com/en-gb/services/scheduler/

_Marak_ 9y ago

I think hook.io can do this. see: http://hook.io/cron http://hook.io/cron

mani-smhack 9y ago

Basically what i am looking is a quartz scheduler alternative with monitoring and admin UI(preferrably hosted) More like sendgrid.com for scheduling

sahilagg06 9y ago

https://github.com/flipkart-incubator/simpleJobScheduler https://github.com/flipkart-incubator/simpleJobScheduler

ako 9y ago

Redwood software RunMyJob: https://www.redwood.com/runmyjobs/ https://www.redwood.com/runmyjobs/

egamirorrim 9y ago

I'm surprised no one's mentioned Chronos - it takes a butt load of setup because it runs on top of Mesos (which requires Zookeeper) etc - but if you already have these tools in your stack it's a pretty tidy clustered scheduler: https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d https://medium.com/airbnb-engineering/chronos-a-replacement-... Edit: if you haven't got this stack around though i'd second calls to use a build tool like Jenkins or Rundeck - of course you probably don't want this on the same Jenkins that builds your jobs... so you'd have to set all that up

assafmo 9y ago

webhook.io works great for me. they also have a great free plan

jnbiche 9y ago
webhook.io is now defunct. They either shut it already down or will soon shut it down. I think webtask.io is somewhat similar. Also, iron.io lets you schedule functions using docker, has full API. AWS Lambda also lets you schedule Lambda and has full API.
- assafmo 9y ago
  sorry i meant webtask.io!