5 ms·

Django makes asynchronous jobs very easy through "management commands" - simple but perfect framework for running any kind of job. The sample runs on Heroku - u

Django makes asynchronous jobs very easy through "management commands" - simple but perfect framework for running any kind of job. The sample runs on Heroku - use a job scheduler and run your commands. Alternative is to use something like Celery: http://docs.celeryproject.org/en/latest/index.html http://docs.celeryproject.org/en/latest/index.html

bsaul 11y ago
Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework. Although in my case it wasn't using it with django but with flask ( but i don't think it makes a big difference). The main issues when you start to have a stack built of big independant systems ( such as a db, a message queue, worker processes, and web server) are more related to deployment, monitoring and failover management. But they are really the big issues that justify the choice of a technology instead of another.
- switch007 11y ago
  I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?
  - deleted 11y ago
    [deleted]
  - joelhaasnoot 11y ago
    Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-management-commands/ https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything. As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.
  - xaritas 11y ago
    In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle: - Java / Scala / Clojure - Go - Erlang (add Elixir if you are comfortable with its maturity) Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.
    - shazow 11y ago
      These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different. For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.
  - bsaul 11y ago
    the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution. as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change. not something you'd want to handle with just an async framework.
    - bsaul 11y ago
      the downvote made me look for a more precise example , so look at quartz for a robust enterprise grade lib for jobs and task management ( and its .net alternative). i suppose celery is, on the paper, comparable in terms of fearures, yet the one time i had to use it made me feel like playing with a brittle toy, and in the end, i had to use a "recycle on every jobs" setting to prevent memory leaks from crashing everything. not that i believe celery is leaking memory, but that tying all the pieces together is a real pita.
- andybak 11y ago
  I suspect people jump to Celery when something much simpler would probably be enough. I've seen people use Celery with all the associated deployment and monitoring complexity when all they wanted to do was run a slow task without blocking a request. You could simply use this: if all you need is to run a slow task and not block a request there's this: https://github.com/defrex/django-after-response https://github.com/defrex/django-after-response I've usually used the system crontab (sometimes via https://github.com/andybak/django-cron https://github.com/andybak/django-cron ). If you really really need a proper task queue then Huey is much lighter than Celery: https://huey.readthedocs.org/en/latest/ https://huey.readthedocs.org/en/latest/ If you do need something more than
  - StavrosK 11y ago
    django-after-response looks fantastic, it'll make 50% of my apps much easier. Does anyone know what the lifecycle of the spawned threads is like? I'm guessing it depends on the server, but it looks like request + tasks need to take less than 30 seconds total in order to not time out.
    - defrex 11y ago
      Ya, it depends on the upstream timeout for the process. I'm pretty sure to default for gunicorn is 30 seconds. After that the process will be killed, which kills the thread.
- jMyles 11y ago
  > celery is really when i started to feel the pain of using python rather than a more enterprise-ready The hendrix project - and its crosstown_traffic API specifically, is designed to alleviate this pain and remind that python (well, Twisted) is absolutely an enterprise-ready asynchronous network solution. http://hendrix.readthedocs.org/ http://hendrix.readthedocs.org/
- ris 11y ago
  "Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework" Say what? The suckiness of celery doesn't come from its use of python. I'd be interested to know how you consider python not to be "enterprise ready".
  - true_religion 11y ago
    I'm not sure what the suckiness of celery is. We've been running it 4 years in production with no problems. It's really a nicely performant option with nice fail over options, and great debugging.

bsaul 11y ago

Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework. Although in my case it wasn't using it with django but with flask ( but i don't think it makes a big difference). The main issues when you start to have a stack built of big independant systems ( such as a db, a message queue, worker processes, and web server) are more related to deployment, monitoring and failover management. But they are really the big issues that justify the choice of a technology instead of another.

switch007 11y ago
I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?
- deleted 11y ago
  [deleted]
- joelhaasnoot 11y ago
  Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-management-commands/ https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything. As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.
- xaritas 11y ago
  In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle: - Java / Scala / Clojure - Go - Erlang (add Elixir if you are comfortable with its maturity) Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.
  - shazow 11y ago
    These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different. For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.
- bsaul 11y ago
  the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution. as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change. not something you'd want to handle with just an async framework.
  - bsaul 11y ago
    the downvote made me look for a more precise example , so look at quartz for a robust enterprise grade lib for jobs and task management ( and its .net alternative). i suppose celery is, on the paper, comparable in terms of fearures, yet the one time i had to use it made me feel like playing with a brittle toy, and in the end, i had to use a "recycle on every jobs" setting to prevent memory leaks from crashing everything. not that i believe celery is leaking memory, but that tying all the pieces together is a real pita.
andybak 11y ago
I suspect people jump to Celery when something much simpler would probably be enough. I've seen people use Celery with all the associated deployment and monitoring complexity when all they wanted to do was run a slow task without blocking a request. You could simply use this: if all you need is to run a slow task and not block a request there's this: https://github.com/defrex/django-after-response https://github.com/defrex/django-after-response I've usually used the system crontab (sometimes via https://github.com/andybak/django-cron https://github.com/andybak/django-cron ). If you really really need a proper task queue then Huey is much lighter than Celery: https://huey.readthedocs.org/en/latest/ https://huey.readthedocs.org/en/latest/ If you do need something more than
- StavrosK 11y ago
  django-after-response looks fantastic, it'll make 50% of my apps much easier. Does anyone know what the lifecycle of the spawned threads is like? I'm guessing it depends on the server, but it looks like request + tasks need to take less than 30 seconds total in order to not time out.
  - defrex 11y ago
    Ya, it depends on the upstream timeout for the process. I'm pretty sure to default for gunicorn is 30 seconds. After that the process will be killed, which kills the thread.
jMyles 11y ago
> celery is really when i started to feel the pain of using python rather than a more enterprise-ready The hendrix project - and its crosstown_traffic API specifically, is designed to alleviate this pain and remind that python (well, Twisted) is absolutely an enterprise-ready asynchronous network solution. http://hendrix.readthedocs.org/ http://hendrix.readthedocs.org/
ris 11y ago
"Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework" Say what? The suckiness of celery doesn't come from its use of python. I'd be interested to know how you consider python not to be "enterprise ready".
- true_religion 11y ago
  I'm not sure what the suckiness of celery is. We've been running it 4 years in production with no problems. It's really a nicely performant option with nice fail over options, and great debugging.