5 ms·
Hey everyone. The idea here is take advantage of the new async/await ORM features in Django 4.1* to build a lightweight background task queue for Django that ru
by drpancake 4y ago
Hey everyone. The idea here is take advantage of the new async/await ORM features in Django 4.1* to build a lightweight background task queue for Django that runs in a single worker process without any external dependencies (i.e. no Redis/RabbitMQ).
It's pretty basic and experimental. Feedback and PRs welcome!
*The Django ORM is still using sync-to-async compatibility layer behind the scenes but they plan to phase this out in future versions.
- canadiantim 4y agoThis looks awesome! I'll definitely check it out. Always on the lookout for minimalist django libraries. I'll give it a go in a couple weeks. Thanks for making it and sharing!
- drpancake 4y agoDo let me know how it goes :)
- xoudini 4y ago> without any external dependencies (i.e. no Redis/RabbitMQ) You still depend on a database with the `Task` model. This would be a no-go for that reason, since there's no reasonable way to have an impact on its behaviour, outside of creating a custom database router to avoid having every third-party library hitting the same database as core logic. If you absolutely must use a model, take a look at enumeration types[1] for a slightly "neater" way to declare choices. [1]: https://docs.djangoproject.com/en/4.1/ref/models/fields/#enumeration-types https://docs.djangoproject.com/en/4.1/ref/models/fields/#enu...
- drpancake 4y agoSure but for 99.9% of Django projects you safely assume there is a database available. I'm not entirely sure why that's a no-go.
- adparadox 4y agoI love the simplicity of this idea because for lots of sites the database works just fine as a queue backend and it reduces the amount of infrastructure needed. I currently use https://github.com/dabapps/django-db-queue https://github.com/dabapps/django-db-queue for https://devmarks.io https://devmarks.io which also uses the database to store tasks instead of a dedicated queue infrastructure. `Django Q` also has an option to use the database, but I haven't tested it at all: https://django-q.readthedocs.io/en/latest/configure.html#orm https://django-q.readthedocs.io/en/latest/configure.html#orm. And if you are already running `redis` for your site, https://github.com/rq/django-rq https://github.com/rq/django-rq is another option. The one benefit of this package is that it is async-first which will be beneficial as Django continually adds in more async capabilities. Nice work! I'm looking forward to trying this out and seeing how it works!
- drpancake 4y agoThanks! I wasn't aware of django-db-queue. As far as I can tell it runs in a single process so I guess running tasks will block any others from running. That's probably acceptable for some use cases. Interesting that django-q has an ORM broker mode. Looks like it forks workers using multiprocessing under-the-hood.
- phoebefactorial 4y agoI've been very excited to see Django embrace asyncio, and your package looks like a great way to do async task queues. I've built a lot of very low traffic Django sites that are all behind user login for only a few users (think company internal-only CRUD tooling) and being able to use a task queue without having to set up a Redis thing is a big bonus for me. Excited to see how this evolves, thank you for sharing it!
- drpancake 4y agoExactly. For many projects it's overkill to set up an external broker like Redis or RabbitMQ. That's where I think Chard fits in.