7 ms·

Just looked at the summary, but it doesn't seem to include a chapter on asynchronous jobs ( such as report creations, emailing, or any kind of long running task

by bsaul 12y ago

Just looked at the summary, but it doesn't seem to include a chapter on asynchronous jobs ( such as report creations, emailing, or any kind of long running tasks). Along with websockets / push architecture, this is often when website designs start to be really complex.

anhtran 12y ago
The issues in sample project are regular challenges. I also think it's not a complex project without real-time parts, searching, async and concurrency tasks. Especially in Python 3.
joelhaasnoot 12y ago
Django makes asynchronous jobs very easy through "management commands" - simple but perfect framework for running any kind of job. The sample runs on Heroku - use a job scheduler and run your commands. Alternative is to use something like Celery: http://docs.celeryproject.org/en/latest/index.html http://docs.celeryproject.org/en/latest/index.html
- bsaul 12y ago
  Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework. Although in my case it wasn't using it with django but with flask ( but i don't think it makes a big difference). The main issues when you start to have a stack built of big independant systems ( such as a db, a message queue, worker processes, and web server) are more related to deployment, monitoring and failover management. But they are really the big issues that justify the choice of a technology instead of another.
  - switch007 12y ago
    I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?
    - deleted 12y ago
      [deleted]
    - joelhaasnoot 12y ago
      Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-management-commands/ https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything. As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.
    - xaritas 12y ago
      In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle: - Java / Scala / Clojure - Go - Erlang (add Elixir if you are comfortable with its maturity) Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.
      - shazow 12y ago
        These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different. For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.
    - bsaul 12y ago
      the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution. as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change. not something you'd want to handle with just an async framework.
  - andybak 12y ago
    I suspect people jump to Celery when something much simpler would probably be enough. I've seen people use Celery with all the associated deployment and monitoring complexity when all they wanted to do was run a slow task without blocking a request. You could simply use this: if all you need is to run a slow task and not block a request there's this: https://github.com/defrex/django-after-response https://github.com/defrex/django-after-response I've usually used the system crontab (sometimes via https://github.com/andybak/django-cron https://github.com/andybak/django-cron ). If you really really need a proper task queue then Huey is much lighter than Celery: https://huey.readthedocs.org/en/latest/ https://huey.readthedocs.org/en/latest/ If you do need something more than
    - StavrosK 12y ago
      django-after-response looks fantastic, it'll make 50% of my apps much easier. Does anyone know what the lifecycle of the spawned threads is like? I'm guessing it depends on the server, but it looks like request + tasks need to take less than 30 seconds total in order to not time out.
      - defrex 12y ago
        Ya, it depends on the upstream timeout for the process. I'm pretty sure to default for gunicorn is 30 seconds. After that the process will be killed, which kills the thread.
  - jMyles 12y ago
    > celery is really when i started to feel the pain of using python rather than a more enterprise-ready The hendrix project - and its crosstown_traffic API specifically, is designed to alleviate this pain and remind that python (well, Twisted) is absolutely an enterprise-ready asynchronous network solution. http://hendrix.readthedocs.org/ http://hendrix.readthedocs.org/
  - ris 12y ago
    "Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework" Say what? The suckiness of celery doesn't come from its use of python. I'd be interested to know how you consider python not to be "enterprise ready".
    - true_religion 12y ago
      I'm not sure what the suckiness of celery is. We've been running it 4 years in production with no problems. It's really a nicely performant option with nice fail over options, and great debugging.

by bsaul 12y ago

anhtran 12y ago

The issues in sample project are regular challenges. I also think it's not a complex project without real-time parts, searching, async and concurrency tasks. Especially in Python 3.

joelhaasnoot 12y ago

Django makes asynchronous jobs very easy through "management commands" - simple but perfect framework for running any kind of job. The sample runs on Heroku - use a job scheduler and run your commands. Alternative is to use something like Celery: http://docs.celeryproject.org/en/latest/index.html http://docs.celeryproject.org/en/latest/index.html

bsaul 12y ago
Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework. Although in my case it wasn't using it with django but with flask ( but i don't think it makes a big difference). The main issues when you start to have a stack built of big independant systems ( such as a db, a message queue, worker processes, and web server) are more related to deployment, monitoring and failover management. But they are really the big issues that justify the choice of a technology instead of another.
- switch007 12y ago
  I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?
  - deleted 12y ago
    [deleted]
  - joelhaasnoot 12y ago
    Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-management-commands/ https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything. As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.
  - xaritas 12y ago
    In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle: - Java / Scala / Clojure - Go - Erlang (add Elixir if you are comfortable with its maturity) Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.
    - shazow 12y ago
      These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different. For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.
  - bsaul 12y ago
    the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution. as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change. not something you'd want to handle with just an async framework.
- andybak 12y ago
  I suspect people jump to Celery when something much simpler would probably be enough. I've seen people use Celery with all the associated deployment and monitoring complexity when all they wanted to do was run a slow task without blocking a request. You could simply use this: if all you need is to run a slow task and not block a request there's this: https://github.com/defrex/django-after-response https://github.com/defrex/django-after-response I've usually used the system crontab (sometimes via https://github.com/andybak/django-cron https://github.com/andybak/django-cron ). If you really really need a proper task queue then Huey is much lighter than Celery: https://huey.readthedocs.org/en/latest/ https://huey.readthedocs.org/en/latest/ If you do need something more than
  - StavrosK 12y ago
    django-after-response looks fantastic, it'll make 50% of my apps much easier. Does anyone know what the lifecycle of the spawned threads is like? I'm guessing it depends on the server, but it looks like request + tasks need to take less than 30 seconds total in order to not time out.
    - defrex 12y ago
      Ya, it depends on the upstream timeout for the process. I'm pretty sure to default for gunicorn is 30 seconds. After that the process will be killed, which kills the thread.
- jMyles 12y ago
  > celery is really when i started to feel the pain of using python rather than a more enterprise-ready The hendrix project - and its crosstown_traffic API specifically, is designed to alleviate this pain and remind that python (well, Twisted) is absolutely an enterprise-ready asynchronous network solution. http://hendrix.readthedocs.org/ http://hendrix.readthedocs.org/
- ris 12y ago
  "Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework" Say what? The suckiness of celery doesn't come from its use of python. I'd be interested to know how you consider python not to be "enterprise ready".
  - true_religion 12y ago
    I'm not sure what the suckiness of celery is. We've been running it 4 years in production with no problems. It's really a nicely performant option with nice fail over options, and great debugging.