9 ms·

Schedule – Python job scheduling for humans

62 points by trueduke 11y ago

tluyben2 11y ago
My team likes Celery to do stuff like this always which is indeed better/necessary for tasks which need to be executed always even when web server dies/restarts. For a lot of jobs that is not needed (where the background is directly related to the foreground and without the foreground process working properly the background processes make no sense anymore) however and then Celery is a pain in the ... to setup & maintain properly and, frankly, overkill, even with the simple setup solutions like Redis. I would recommend in process schedulers for those cases as they require no setup.
arsalanb 11y ago
Can this be used in production?
- est 11y ago
  You'll need to wrap it in `threading` to avoid block your main thread, or use greenlet, or wrap it as a RESTful service, then you have the problem of high-availability, so a distributed shared consensus system would be better, the next thing you know you have NTP issues... The problem goes on.
- bpicolo 11y ago
  Job scheduling not really rocket science, so sure. Job scheduling when things fail is harder
StavrosK 11y ago
I've been talking with a few developer friends about libraries lately, and this one has made me realize what they're saying. As Python developers, DSLs are completely not in our DNA, but this library strikes me as something that would look much more Pythonic with a DSL. Wouldn't it be better to have descriptions of: Every day at 10:30 do myfunc Every Monday do otherfunc Every hour do whatever Rather than the current .every().day.at("10:30").do(myfunc)?
- gvalkov 11y ago
  I find it hard to appreciate these inventive types of API design. Something more traditional and boring, like Celery's crontab[1] or dateutils rrule[2] seem quite sufficient. For example: crontab(hour=18, minute=30, day_of_week='*') [1]: http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#crontab-schedules http://celery.readthedocs.org/en/latest/userguide/periodic-t... [2]: https://labix.org/python-dateutil#head-470fa22b2db72000d7abe698a5783a46b0731b57 https://labix.org/python-dateutil#head-470fa22b2db72000d7abe...
  - StavrosK 11y ago
    I'm not sure why, given two equally flexible designs, we should choose the least expressive. The fact that cron is sufficient doesn't mean I don't still have to google how to run things every three hours every second Monday, for example.
- draegtun 11y ago
  I agree. This scheduler dialect for Rebol is pretty much the DSL you describe - http://softinnov.org/rebol/scheduler.shtml http://softinnov.org/rebol/scheduler.shtml
- anentropic 11y ago
  I'd rather have neither. I particularly dislike the .every().day.at("10:30").do(myfunc) thing where methods and attributes are abused in order to artificially make it read like a sentence. This is not Pythonic at all <troll>it's like some childish Ruby code</troll>
  - StavrosK 11y ago
    That's not a very constructive reply.
  - omni 11y ago
    What specifically do you think makes this un-Pythonic? It's very readable.
vas1 11y ago
Currently I use APScheduler for this exact use-case. This looks interesting and I will be taking a closer look at it.
- Drdrdrq 11y ago
  Thank you, looks really nice and customizable! Did you have any issues with it?
  - hotdox 11y ago
    Not the author of previous comment, but my main problem is inability to limit time of job execution in APScheduler.
St-Clock 11y ago
We prefer to use Celery to schedule periodic tasks [1] and are moving most of our cron jobs to Celery because it is easier to monitor, test, retry, and send reports on failed jobs. We extended [2] django-celery db backend to store enough information about the periodic tasks to retry them directly in Django admin or to filter the results (we support storing the task results in JSON instead of pickle). Celery is probably more difficult to configure but is far more flexible and robust than this lib... [1] http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html http://celery.readthedocs.org/en/latest/userguide/periodic-t... [2] https://github.com/resulto-admin/django-celery-fulldbresult https://github.com/resulto-admin/django-celery-fulldbresult
mpdehaan2 11y ago
Aside: I'm not a fan of the "for humans" label that seems to be common among some Python projects now. It implies the other tools are possibly designed by monkeys, to a degree, which is often totally not the case. As a particular example, while some libs are "friendly", they are also limited. In this case, celery is implied to be not "for humans", but it actually offers distributed job queuing and jobs that can survive a reboot, which is pretty darn important. I'm not going to name names of other libs doing this, but there are a few that have picked API simplicity over complete feature set. This is obviously a balance - BUT I think it unfairly casts the alternatively libs in a questionable light, and often they can do a lot more. FWIW, the Ruby-design into the method space can also be reduced if desired to make this a bit more Pythonic (though some things are personal preferences): scheduler = schedule.scheduler() scheduler(callback=job).every(minutes=10) scheduler(callback=job).every(day=1).at(hour=10, minutes=30) Having functions like "Wednesday" or "minutes" complicate programatic usage, where in the above, it would be possible to pass in a constant. One thing I also notice (not sure if Ruby inspired or not), is that it's keeping global state in the module (see run_pending) rather than having different scheduler objects. This somewhat limits reusability by having this embedded magic and not being able to isolate schedulers.
- maxerickson 11y ago
  The global scheduler object is provided for convenience, it is possible to create others. I agree about the for humans branding. I also don't find this api very appealing, but I haven't given much consideration to what an interval api should look like.
  - rch 11y ago
    It should probably look a lot like ISO 8601.
- icebraining 11y ago
  Monkeys? Where did that come from? The idea of "for humans" just means that the UI was developed with an humane approach - as opposed to an undesigned UI, or one designed merely for the simplicity of the implementation, and that doesn't take into consideration things like our need for consistency to make up for a non-perfect memory. That said, I personally don't think Celery suffers from this problem, but API that was being replaced by the first project to used this moniker - urllib2 - definitively did.
mckoss 11y ago
I personally dislike this style of "fluent api". It is not more "for humans" just because it reads as English sentences. In fact, it can be quite difficult to reason about. What, for example is "schedule. every(10)"? By itself, it does not seem to represent a useful concept other than a "partial" schedule statement. I would prefer to use an api using keyword arguments to set the various (optional) parameters to build a schedule.
- Drdrdrq 11y ago
  Exactly! One example of this is fudge... Annoying to use, especially when docs are less than perfect.
AnthonBerg 11y ago
I like the feel of it and the thought behind it, but ... while true, sleep? I get very uncomfortable with any kind of polling, especially polling timers for checking for jobs with timespans between them. Maybe it's just me?
- est 11y ago
  every cron or cron-like service has some kind of polling behind. Beanstalk, redis-queue, you name it. Many open source redis based scheduler are all implemented in a sleep(5) loop. Even the Unix cron is implemented with a sleep() call https://en.wikipedia.org/wiki/Cron#History https://en.wikipedia.org/wiki/Cron#History
- drudru11 11y ago
  As another commentor noted, this is the way every system is implemented. It is simple, slightly inefficient, and deals with most of the nasty problems that arise when dealing with time on all systems today.
arthurcolle 11y ago
Why is the time.sleep(1) needed inside the while loop?
- est 11y ago
  Because that's one second resolution of the scheduler.
- Drdrdrq 11y ago
  Otherwise app would burn CPU like crazy...
drudru11 11y ago
So - this is interesting. The thing that I always look for is how the system deals with all of the time issues that come up. For example, how does it deal with Daylight Savings, or a leap second, or adjustments to time, or the daemon getting restarted, etc. I don't see anything specific to this library that handles these scenarios. Is there any system out there that has a good design for handling these issues?
lifeisstillgood 11y ago
My issue is how to make this distributed (unless Inmissed that) Have many out of process (on disk) files that contain some or all of the phrases "start package a with Params b at time c" And have the scheduler pass over the actual running to another process - possibly another machine entirely Would like to get my teeth into that

62 points by trueduke 11y ago

My team likes Celery to do stuff like this always which is indeed better/necessary for tasks which need to be executed always even when web server dies/restarts. For a lot of jobs that is not needed (where the background is directly related to the foreground and without the foreground process working properly the background processes make no sense anymore) however and then Celery is a pain in the ... to setup & maintain properly and, frankly, overkill, even with the simple setup solutions like Redis. I would recommend in process schedulers for those cases as they require no setup.

arsalanb 11y ago

Can this be used in production?

est 11y ago
You'll need to wrap it in `threading` to avoid block your main thread, or use greenlet, or wrap it as a RESTful service, then you have the problem of high-availability, so a distributed shared consensus system would be better, the next thing you know you have NTP issues... The problem goes on.
bpicolo 11y ago
Job scheduling not really rocket science, so sure. Job scheduling when things fail is harder

StavrosK 11y ago

I've been talking with a few developer friends about libraries lately, and this one has made me realize what they're saying. As Python developers, DSLs are completely not in our DNA, but this library strikes me as something that would look much more Pythonic with a DSL. Wouldn't it be better to have descriptions of: Every day at 10:30 do myfunc Every Monday do otherfunc Every hour do whatever Rather than the current .every().day.at("10:30").do(myfunc)?

gvalkov 11y ago
I find it hard to appreciate these inventive types of API design. Something more traditional and boring, like Celery's crontab[1] or dateutils rrule[2] seem quite sufficient. For example: crontab(hour=18, minute=30, day_of_week='*') [1]: http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#crontab-schedules http://celery.readthedocs.org/en/latest/userguide/periodic-t... [2]: https://labix.org/python-dateutil#head-470fa22b2db72000d7abe698a5783a46b0731b57 https://labix.org/python-dateutil#head-470fa22b2db72000d7abe...
- StavrosK 11y ago
  I'm not sure why, given two equally flexible designs, we should choose the least expressive. The fact that cron is sufficient doesn't mean I don't still have to google how to run things every three hours every second Monday, for example.
draegtun 11y ago
I agree. This scheduler dialect for Rebol is pretty much the DSL you describe - http://softinnov.org/rebol/scheduler.shtml http://softinnov.org/rebol/scheduler.shtml
anentropic 11y ago
I'd rather have neither. I particularly dislike the .every().day.at("10:30").do(myfunc) thing where methods and attributes are abused in order to artificially make it read like a sentence. This is not Pythonic at all <troll>it's like some childish Ruby code</troll>
- StavrosK 11y ago
  That's not a very constructive reply.
- omni 11y ago
  What specifically do you think makes this un-Pythonic? It's very readable.

vas1 11y ago

Currently I use APScheduler for this exact use-case. This looks interesting and I will be taking a closer look at it.

Drdrdrq 11y ago
Thank you, looks really nice and customizable! Did you have any issues with it?
- hotdox 11y ago
  Not the author of previous comment, but my main problem is inability to limit time of job execution in APScheduler.

St-Clock 11y ago

We prefer to use Celery to schedule periodic tasks [1] and are moving most of our cron jobs to Celery because it is easier to monitor, test, retry, and send reports on failed jobs. We extended [2] django-celery db backend to store enough information about the periodic tasks to retry them directly in Django admin or to filter the results (we support storing the task results in JSON instead of pickle). Celery is probably more difficult to configure but is far more flexible and robust than this lib... [1] http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html http://celery.readthedocs.org/en/latest/userguide/periodic-t... [2] https://github.com/resulto-admin/django-celery-fulldbresult https://github.com/resulto-admin/django-celery-fulldbresult

mpdehaan2 11y ago

Aside: I'm not a fan of the "for humans" label that seems to be common among some Python projects now. It implies the other tools are possibly designed by monkeys, to a degree, which is often totally not the case. As a particular example, while some libs are "friendly", they are also limited. In this case, celery is implied to be not "for humans", but it actually offers distributed job queuing and jobs that can survive a reboot, which is pretty darn important. I'm not going to name names of other libs doing this, but there are a few that have picked API simplicity over complete feature set. This is obviously a balance - BUT I think it unfairly casts the alternatively libs in a questionable light, and often they can do a lot more. FWIW, the Ruby-design into the method space can also be reduced if desired to make this a bit more Pythonic (though some things are personal preferences): scheduler = schedule.scheduler() scheduler(callback=job).every(minutes=10) scheduler(callback=job).every(day=1).at(hour=10, minutes=30) Having functions like "Wednesday" or "minutes" complicate programatic usage, where in the above, it would be possible to pass in a constant. One thing I also notice (not sure if Ruby inspired or not), is that it's keeping global state in the module (see run_pending) rather than having different scheduler objects. This somewhat limits reusability by having this embedded magic and not being able to isolate schedulers.

maxerickson 11y ago
The global scheduler object is provided for convenience, it is possible to create others. I agree about the for humans branding. I also don't find this api very appealing, but I haven't given much consideration to what an interval api should look like.
- rch 11y ago
  It should probably look a lot like ISO 8601.
icebraining 11y ago
Monkeys? Where did that come from? The idea of "for humans" just means that the UI was developed with an humane approach - as opposed to an undesigned UI, or one designed merely for the simplicity of the implementation, and that doesn't take into consideration things like our need for consistency to make up for a non-perfect memory. That said, I personally don't think Celery suffers from this problem, but API that was being replaced by the first project to used this moniker - urllib2 - definitively did.

mckoss 11y ago

I personally dislike this style of "fluent api". It is not more "for humans" just because it reads as English sentences. In fact, it can be quite difficult to reason about. What, for example is "schedule. every(10)"? By itself, it does not seem to represent a useful concept other than a "partial" schedule statement. I would prefer to use an api using keyword arguments to set the various (optional) parameters to build a schedule.

Drdrdrq 11y ago
Exactly! One example of this is fudge... Annoying to use, especially when docs are less than perfect.

AnthonBerg 11y ago

I like the feel of it and the thought behind it, but ... while true, sleep? I get very uncomfortable with any kind of polling, especially polling timers for checking for jobs with timespans between them. Maybe it's just me?

est 11y ago
every cron or cron-like service has some kind of polling behind. Beanstalk, redis-queue, you name it. Many open source redis based scheduler are all implemented in a sleep(5) loop. Even the Unix cron is implemented with a sleep() call https://en.wikipedia.org/wiki/Cron#History https://en.wikipedia.org/wiki/Cron#History
drudru11 11y ago
As another commentor noted, this is the way every system is implemented. It is simple, slightly inefficient, and deals with most of the nasty problems that arise when dealing with time on all systems today.

arthurcolle 11y ago

Why is the time.sleep(1) needed inside the while loop?

est 11y ago
Because that's one second resolution of the scheduler.
Drdrdrq 11y ago
Otherwise app would burn CPU like crazy...

drudru11 11y ago

So - this is interesting. The thing that I always look for is how the system deals with all of the time issues that come up. For example, how does it deal with Daylight Savings, or a leap second, or adjustments to time, or the daemon getting restarted, etc. I don't see anything specific to this library that handles these scenarios. Is there any system out there that has a good design for handling these issues?

lifeisstillgood 11y ago

My issue is how to make this distributed (unless Inmissed that) Have many out of process (on disk) files that contain some or all of the phrases "start package a with Params b at time c" And have the scheduler pass over the actual running to another process - possibly another machine entirely Would like to get my teeth into that