4 ms·

Not sure what you mean by logs, resources and changing trees. But there is logging support (the python logging module). Resources: There is AMQP QoS which make

by asksol 17y ago

Not sure what you mean by logs, resources and changing trees. But there is logging support (the python logging module). Resources: There is AMQP QoS which makes sure it only receives as many tasks as it can handle. Task hard and soft time-limits is coming in 1.0 (patch ready). If the soft timeout is exceeded an exception is raised which the task can catch to do any clean up before the hard time limit is exceeded and the task is forcefully killed. Rate limit (per task type or global) using the token bucket algorithm (which allows for bursts of data). For 1.0 (patch ready and tested) Otherwise you have to OS process resource limits (cpu/memory etc). Monitoring is coming in 1.0 as well, someone is working on a monitoring system with a web-frontend where you can see the current state of the system (support for deleting already published tasks might be added, but then on an opt-in basis) The current scheduling system is flawed (it uses the database, which is a dead end in my opinion), a new solution is almost ready which uses a separate centralized service that works like a clock sending out messages at schedule time: http://wiki.github.com/ask/celery/rewriting-the-periodic-task-service http://wiki.github.com/ask/celery/rewriting-the-periodic-tas... Now for changing trees, I'm not sure what you mean here, please correct me if I misunderstood. Messages can not be changed once they have been published, so the task itself is responsible for changing the execution order. You can chain tasks, so say TaskA launches another task. You can retry tasks if they fail. Oh and there's the message routing features made available by AMQP, which means you can have different servers/instances handle different tasks. Celery has a lot of features, and even more is under development, so I don't think I can list them all here. I can't see anything hindering a Celery implementation of Norc, but as I read it you started working on this before celery started. Bad luck when we could have shared a lot of work :(

darrellsilver 17y ago
That sounds super interesting! This sentence raises a question: >You can chain tasks, so say TaskA launches another task. You can retry tasks if they fail. How does this chaining work? Does each Task define its children? I've found that its cleaner to separate tasks from their place in the tree. For example, a script that downloads a CSV file each hour shouldn't care how that file is used.
- asksol 17y ago
  There's nothing built in to celery for this, but some people have already made different solutions (one which may be added as standard in the future). There's a ChainableTask where you can chain tasks together, the next running when the previous is finished, and they can optionally take the result of the previous task as a parameter.

by asksol 17y ago

darrellsilver 17y ago

That sounds super interesting! This sentence raises a question: >You can chain tasks, so say TaskA launches another task. You can retry tasks if they fail. How does this chaining work? Does each Task define its children? I've found that its cleaner to separate tasks from their place in the tree. For example, a script that downloads a CSV file each hour shouldn't care how that file is used.

asksol 17y ago
There's nothing built in to celery for this, but some people have already made different solutions (one which may be added as standard in the future). There's a ChainableTask where you can chain tasks together, the next running when the previous is finished, and they can optionally take the result of the previous task as a parameter.