4 ms·

Here is a question: I often find myself wanting to do background tasks within my application (say, a Flask app) without running celeryd. While celery is quite u

• JimmyL 13y ago
  I've read about https://github.com/tthieman/dagobah https://github.com/tthieman/dagobah recently, but I haven't used it...also, if you think celery is too heavy, you'll probably think this is too. I've also heard good things about https://github.com/binarydud/pyres https://github.com/binarydud/pyres. Despite all that, I usually just find myself running celery. I'll usually have a redis instance going anyways that I can use as a sub-optimal queue, and if it's a small enough site, I'll run a single worker instance on the same machine as the host.
• scott_w 13y ago
  Celery can use a database backend, such as Postgres. Maybe it can work with sqlite?
  • shuzchen 13y ago
    You definitely can use celery with sqlite via the sqlalchemy backend, but since rdbms' aren't very good task queues there are drawbacks: see http://docs.celeryproject.org/en/latest/getting-started/brokers/sqlalchemy.html http://docs.celeryproject.org/en/latest/getting-started/brok...
    • scott_w 13y ago
      That's true. However, as an answer to the above it may well be acceptable without needing a different library.
• whitehat2k9 13y ago
  Anyone want to chime in on why Python threads and subprocesses can't be used?
  • wting 13y ago
    Subprocesses aren't used because the cost of forking is high. It's more efficient to fork a worker once, then have that worker keep doing work. The problem now becomes transferring data to that worker, which is where celery comes into play. Python threads cannot be used for CPU heavy loads because they all run on a single core.[0][1] [0]: CPython threads are kernel threads but effectively act like user threads. All threads run within a single VM and thus restricted to a single core. [1]: Jython does not have the same limitation.
    • m0th87 13y ago
      IIRC, unless you explicitly state otherwise, Celery forks for every task. A lot of task queues do this because it becomes a lot easier to recovery from errors. Multiprocessing makes it pretty straight-forward to manage inter-process communication with queues, but you'd have to write all the other stuff task queues handle for you, such as error management.
• oxtopus 13y ago
  Running your Flask application with uWSGI could allow you to do just that by way of the spool and other related decorators: http://uwsgi-docs.readthedocs.org/en/latest/PythonDecorators.html http://uwsgi-docs.readthedocs.org/en/latest/PythonDecorators...
  • mh- 13y ago
    +1 for uWSGI. The Spooler is awesome and also, if you need lower-level stuff, have a look at its SharedArea and Queue[1] frameworks, as well as Mules[2]. [1] http://uwsgi-docs.readthedocs.org/en/latest/Queue.html http://uwsgi-docs.readthedocs.org/en/latest/Queue.html [2] http://uwsgi-docs.readthedocs.org/en/latest/Mules.html http://uwsgi-docs.readthedocs.org/en/latest/Mules.html
• ddorian43 13y ago
  Advanced Python Scheduler (APScheduler) is a light but powerful in-process task scheduler that lets you schedule functions (or any other python callables) to be executed at times of your choosing. This can be a far better alternative to externally run cron scripts for long-running applications (e.g. web applications), as it is platform neutral and can directly access your application’s variables and functions.
• cpenner461 13y ago
  I've been using Python RQ (http://python-rq.org/ http://python-rq.org/) as a lightweight alternative to celery. It uses Redis as its backend so I guess that might not meet the "SQLite" qualification, but has been light weight enough for my application (e.g. no reqt for a message queue like rabbit). Really liking it so far.
  • m0th87 13y ago
    We're using RQ in production and are very happy with it. We chose it over Celery because the "protocol" is much more straight-forward than Celery - when something fails, I can peek at the state of redis or the RQ source code and know what's going on. The only downside is that, like Celery, RQ forks for every task. This can feel a bit heavy for some of our simpler tasks, although it hasn't proven to be an issue yet.
• ewheeler 13y ago
  APScheduler is a nice in-process alternative that doesn't require running another process or using a db for persistence. Quick to start using and no external dependencies. https://pypi.python.org/pypi/APScheduler/ https://pypi.python.org/pypi/APScheduler/
• Splotzenbobble 13y ago
  I just spent a whole day figuring out and setting up celery and a broker just to have a small maintenance task run for a few milliseconds about once an hour in a django application. This seems needlessly heavy and complex, and somewhat negated the development time I expected to save by basing this project on python rather than java. I'm not blaming python, but rather my knowledge of it. There must be an easier way to run small periodic scheduled tasks in the background than this.
• crdoconnor 13y ago
  >While celery is quite useful, I just want to spin up an application and have it Just Work without external dependencies. With celery, now I have to run celeryd, run some sort of queue (redis, rabbitmq, etc), and just worry about all of these pieces simply to execute long-running background tasks. You can use redis as a backend for celery (which you should be using already for caching, probably), and AFAIK you can use a database as a backend too (although I haven't tried it). That means that celery is exactly one dependency. You need some kind of persistent store so that it can recover the task queue from hardware failure. But it can probably use the backend you already have.