3 ms·

I'll add that if you need to background anything, django-celery makes it easy. celeryd is also good for scheduling tasks with either single or periodic executi

I'll add that if you need to background anything, django-celery makes it easy. celeryd is also good for scheduling tasks with either single or periodic execution. http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#first-steps http://docs.celeryproject.org/en/latest/getting-started/firs... You can also use it in a typical task queue role. It will pickle or serialize the Python objects that accompany tasks for transmission over the wire.

obviouslygreen 13y ago
Note that periodic execution, one of the more useful aspects of celery, is not supported on all backends. If you're using rabbitMQ, you're golden. If you're using most or all other transports (including SQS), you're likely SOL and will have to resort to either bare cron jobs or something like django-cronograph that accomplishes the same thing. celery is definitely nice, but it does have its limitations (like everything else).
- hiddentaco 13y ago
  I'm really interested in peoples opinion on Celery vs Others when using Django. I'm about to hit a point in my pet project where I need something but I havn't chosen my approach yet. I've ehard Pyres is better than Celery.
  - obviouslygreen 13y ago
    Once you wrap your head around one (embarrassingly, it took me a while to really understand what celery was doing -- I had an easier time getting comfortable with the multiprocessing library), it shouldn't be difficult to pick up another and compare them locally. Celery does what it's supposed to; that's about all I can say for it. I'd suggest writing a few simple jobs and testing them in similar environments under as much load as you can. Look for things like consistency of logging and running tasks exactly once across multiple consumers despite the semantics of the queue you pick.
- crdoconnor 13y ago
  This is good to know. I've just started using it with redis. I never really saw the point of that part of celery anyway. Cron's perfectly adequate on its own (and installed just about everywhere).
  - obviouslygreen 13y ago
    The nice thing about it from our point of view is the ability to keep everything in source control. Granted, you can keep your cron jobs as scripts, but celery+rabbitMQ allows you to keep it all in Python without the extra deployment step. It's not a huge deal, but it does help a bit with maintainability and getting new devs up to speed.
    - crdoconnor 13y ago
      We keep all the deployment stuff in source control anyway. For me, it would just mean shifting it from being declared by salt in the "infrastructure" folder to being written in python in the django folder. I agree that you should keep the code itself in python, but that can be handled just by making it a management command that gets called by cron. Sysadmins understand and love cron, too. I think celery reinventing the wheel might make some of them a bit mad.
  - ludwigvan 13y ago
    I'm using cron instead of celery daemon for those tasks, but here are two advantages: - being able to modify the task frequency from the admin - cron needs special care with setting the environment variables like PATH
wahnfrieden 13y ago
We've had a lot of luck just using gunicorn with a post_response hook for doing some extra work without making the client wait on it before getting a response. It avoids adding another dependency and is pretty straightforward. It makes sense for us since we don't have HUGE tasks that need to be queued, but some things that can take a maybe 0.5 to 3 seconds and where the HTTP response can be sent back before doing this work. I should open source this at some point, it's a small amount of code but pretty handy.
thejosh 13y ago
Using celery for the first time, coming from PHP-land, blew my mind. Python is now my language of choice for "getting shit done".
- crdoconnor 13y ago
  Returning to PHP land afterwards feels like you're taping your code together with duct tape and bits of string.
  - ceol 13y ago
    At the risk of this becoming a "bash PHP" thread, that's exactly how I feel. Unless I'm doing something very simple, it's always easier, cleaner, and simpler to do it in Python. Even building a basic CRUD app is night and day between the two languages; Django is beautiful and intuitive, while the PHP frameworks come close but have a few snags that make working with them frustrating. I write this not to meaninglessly bash PHP but to hopefully convince even a single developer passing by to try out Python. I thought the same thing as you, passer-by dev, once: "Why do I need Python? It's stupid. It uses whitespace weirdly. I can do whatever I need in PHP." But then you realize you could be doing everything you're doing in PHP... better, smarter, and faster.
    - camus 13y ago
      The fact is PHP has excellent frameworks. I tried Django and Flask , and love them. But for now , i'm sticking to PHP( to much legacy code to port ). For new projects ? sure. The only thing which is a bit more complicated with serverside python is deployment on client servers, the rest ( development , testing ), is easier than PHP. I only wished Python had type hinting for functions and methods like PHP does though. Aside from these issues , Python is perfect,more consice and have awesome stuffs like decorators. And a whole lot of awesome libraries.
    - lmm 13y ago
      I'm getting the same feeling now that I have to work in python/django after having used scala. So it's always worth trying new languages, even when you've found one that seems great at the time.
olavgg 13y ago
What about http://pythonhosted.org/APScheduler/ http://pythonhosted.org/APScheduler/ ? Celery is for more massive, high capacity solutions.
dangoldin 13y ago
Yea - that's going to be one of the packages I mention in the follow up. I ended up using RabbitMQ as the backend for it which wasn't that easy to setup so I am curious to see what others are using.

obviouslygreen 13y ago

Note that periodic execution, one of the more useful aspects of celery, is not supported on all backends. If you're using rabbitMQ, you're golden. If you're using most or all other transports (including SQS), you're likely SOL and will have to resort to either bare cron jobs or something like django-cronograph that accomplishes the same thing. celery is definitely nice, but it does have its limitations (like everything else).

hiddentaco 13y ago
I'm really interested in peoples opinion on Celery vs Others when using Django. I'm about to hit a point in my pet project where I need something but I havn't chosen my approach yet. I've ehard Pyres is better than Celery.
- obviouslygreen 13y ago
  Once you wrap your head around one (embarrassingly, it took me a while to really understand what celery was doing -- I had an easier time getting comfortable with the multiprocessing library), it shouldn't be difficult to pick up another and compare them locally. Celery does what it's supposed to; that's about all I can say for it. I'd suggest writing a few simple jobs and testing them in similar environments under as much load as you can. Look for things like consistency of logging and running tasks exactly once across multiple consumers despite the semantics of the queue you pick.
crdoconnor 13y ago
This is good to know. I've just started using it with redis. I never really saw the point of that part of celery anyway. Cron's perfectly adequate on its own (and installed just about everywhere).
- obviouslygreen 13y ago
  The nice thing about it from our point of view is the ability to keep everything in source control. Granted, you can keep your cron jobs as scripts, but celery+rabbitMQ allows you to keep it all in Python without the extra deployment step. It's not a huge deal, but it does help a bit with maintainability and getting new devs up to speed.
  - crdoconnor 13y ago
    We keep all the deployment stuff in source control anyway. For me, it would just mean shifting it from being declared by salt in the "infrastructure" folder to being written in python in the django folder. I agree that you should keep the code itself in python, but that can be handled just by making it a management command that gets called by cron. Sysadmins understand and love cron, too. I think celery reinventing the wheel might make some of them a bit mad.
- ludwigvan 13y ago
  I'm using cron instead of celery daemon for those tasks, but here are two advantages: - being able to modify the task frequency from the admin - cron needs special care with setting the environment variables like PATH

wahnfrieden 13y ago

We've had a lot of luck just using gunicorn with a post_response hook for doing some extra work without making the client wait on it before getting a response. It avoids adding another dependency and is pretty straightforward. It makes sense for us since we don't have HUGE tasks that need to be queued, but some things that can take a maybe 0.5 to 3 seconds and where the HTTP response can be sent back before doing this work. I should open source this at some point, it's a small amount of code but pretty handy.

thejosh 13y ago

Using celery for the first time, coming from PHP-land, blew my mind. Python is now my language of choice for "getting shit done".

crdoconnor 13y ago
Returning to PHP land afterwards feels like you're taping your code together with duct tape and bits of string.
- ceol 13y ago
  At the risk of this becoming a "bash PHP" thread, that's exactly how I feel. Unless I'm doing something very simple, it's always easier, cleaner, and simpler to do it in Python. Even building a basic CRUD app is night and day between the two languages; Django is beautiful and intuitive, while the PHP frameworks come close but have a few snags that make working with them frustrating. I write this not to meaninglessly bash PHP but to hopefully convince even a single developer passing by to try out Python. I thought the same thing as you, passer-by dev, once: "Why do I need Python? It's stupid. It uses whitespace weirdly. I can do whatever I need in PHP." But then you realize you could be doing everything you're doing in PHP... better, smarter, and faster.
  - camus 13y ago
    The fact is PHP has excellent frameworks. I tried Django and Flask , and love them. But for now , i'm sticking to PHP( to much legacy code to port ). For new projects ? sure. The only thing which is a bit more complicated with serverside python is deployment on client servers, the rest ( development , testing ), is easier than PHP. I only wished Python had type hinting for functions and methods like PHP does though. Aside from these issues , Python is perfect,more consice and have awesome stuffs like decorators. And a whole lot of awesome libraries.
  - lmm 13y ago
    I'm getting the same feeling now that I have to work in python/django after having used scala. So it's always worth trying new languages, even when you've found one that seems great at the time.

olavgg 13y ago

What about http://pythonhosted.org/APScheduler/ http://pythonhosted.org/APScheduler/ ? Celery is for more massive, high capacity solutions.

dangoldin 13y ago

Yea - that's going to be one of the packages I mention in the follow up. I ended up using RabbitMQ as the backend for it which wasn't that easy to setup so I am curious to see what others are using.