4 ms·

Schedule: Python job scheduling for humans.

13 points by dbader 13y ago

bifrost 13y ago
I poked through this briefly. Anyone know what advantages this has over built in system utilities like cron/etc aside from being written in python?
- dbader 13y ago
  Using system utilities like cron, can be suboptimal if all you're trying to do is schedule periodic tasks within an application, e.g. database cleanup tasks, periodic polling of other services etc. The main gripes with cron are: - cron can be overkill. It requires maintaining a separate crontab file for your app. Wit schedule you can keep everything in pure Python. - cron is per machine. Multiple app servers require locks of some sort to prevent job duplication. To solve this, we could put triggered jobs on a shared jobqueue (beanstalkc, resque, ...) and have them processed by several workers. - cron's syntax, while powerful, can be difficult to understand and maintain. Schedule provides an easy to read syntax based on the builder pattern, e.g. "schedule.every(2).hours.do(job, args)" Much of this was said in better words by Addam Wiggins: http://adam.heroku.com/past/2010/4/13/rethinking_cron/ http://adam.heroku.com/past/2010/4/13/rethinking_cron/ I wrote schedule because I needed a lightweight scheduling solution for a Python web service backend. It's 'clockwork' (https://github.com/tomykaira/clockwork https://github.com/tomykaira/clockwork) for Python.
jgraham 13y ago
Isn't it quite unfriendly to make the caller implement the main loop, and require periodic polling of the scheduler in order to run jobs? When I have implemented things like this before, I kept tuples of (time_of_next_run, job) in a heap. Then the scheduler had a loop like: while True: now = time.time() while job_heap[0][0] < now: time, job = heapq.heappop(job_heap) job.run() heapq.heappush(job_heap, (job.next_time(), job)) time.sleep(job_heap[0][0] - time.time()) This reduces the amount of polling needed, since you sleep until the next job is ready.
- dbader 13y ago
  You're certainly right that the module should provide a main loop of some sort for convenience. I wasn't totally sure how to implement this to make everyone happy -- some people may want to use time.sleep(), others want threading.Event().wait(). Maybe I'll simply go with something like this: def run_forever(sleep_func=time.sleep): ...
olavgg 13y ago
I was first thinking, why not just use APScheduler. But then I saw the syntax... neat!!
- dbader 13y ago
  Thanks :-)
mbi 13y ago
While the API looks really nice and clean, this seems to lack any sort of task distribution mechanism, i.e. tasks run in-process but cannot be passed off to off-process workers, like in e.g. Celery [1] or RQ [2] (the former also does scheduling, the latter is more of a job queueing library, that can do scheduling via an add-on [3]) 1: http://www.celeryproject.org/ http://www.celeryproject.org/ 2: http://python-rq.org/ http://python-rq.org/ 3: https://github.com/ui/rq-scheduler https://github.com/ui/rq-scheduler
- dbader 13y ago
  I use something like this for submitting jobs to beanstalkc: schedule.every().hour.do(beanstalkc.put, "my_job_id")

13 points by dbader 13y ago

I poked through this briefly. Anyone know what advantages this has over built in system utilities like cron/etc aside from being written in python?

dbader 13y ago
Using system utilities like cron, can be suboptimal if all you're trying to do is schedule periodic tasks within an application, e.g. database cleanup tasks, periodic polling of other services etc. The main gripes with cron are: - cron can be overkill. It requires maintaining a separate crontab file for your app. Wit schedule you can keep everything in pure Python. - cron is per machine. Multiple app servers require locks of some sort to prevent job duplication. To solve this, we could put triggered jobs on a shared jobqueue (beanstalkc, resque, ...) and have them processed by several workers. - cron's syntax, while powerful, can be difficult to understand and maintain. Schedule provides an easy to read syntax based on the builder pattern, e.g. "schedule.every(2).hours.do(job, args)" Much of this was said in better words by Addam Wiggins: http://adam.heroku.com/past/2010/4/13/rethinking_cron/ http://adam.heroku.com/past/2010/4/13/rethinking_cron/ I wrote schedule because I needed a lightweight scheduling solution for a Python web service backend. It's 'clockwork' (https://github.com/tomykaira/clockwork https://github.com/tomykaira/clockwork) for Python.

jgraham 13y ago

Isn't it quite unfriendly to make the caller implement the main loop, and require periodic polling of the scheduler in order to run jobs? When I have implemented things like this before, I kept tuples of (time_of_next_run, job) in a heap. Then the scheduler had a loop like: while True: now = time.time() while job_heap[0][0] < now: time, job = heapq.heappop(job_heap) job.run() heapq.heappush(job_heap, (job.next_time(), job)) time.sleep(job_heap[0][0] - time.time()) This reduces the amount of polling needed, since you sleep until the next job is ready.

dbader 13y ago
You're certainly right that the module should provide a main loop of some sort for convenience. I wasn't totally sure how to implement this to make everyone happy -- some people may want to use time.sleep(), others want threading.Event().wait(). Maybe I'll simply go with something like this: def run_forever(sleep_func=time.sleep): ...

olavgg 13y ago

I was first thinking, why not just use APScheduler. But then I saw the syntax... neat!!

dbader 13y ago
Thanks :-)

mbi 13y ago

While the API looks really nice and clean, this seems to lack any sort of task distribution mechanism, i.e. tasks run in-process but cannot be passed off to off-process workers, like in e.g. Celery [1] or RQ [2] (the former also does scheduling, the latter is more of a job queueing library, that can do scheduling via an add-on [3]) 1: http://www.celeryproject.org/ http://www.celeryproject.org/ 2: http://python-rq.org/ http://python-rq.org/ 3: https://github.com/ui/rq-scheduler https://github.com/ui/rq-scheduler

dbader 13y ago
I use something like this for submitting jobs to beanstalkc: schedule.every().hour.do(beanstalkc.put, "my_job_id")