3 ms·

Thanks for the kind words. I should update the part about the queues. I started using this pattern for SKIP LOCKED and it works well for multiple workers (searc

Thanks for the kind words. I should update the part about the queues. I started using this pattern for SKIP LOCKED and it works well for multiple workers (search engine crawlers). https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock... And implemented in Django: https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_for_update https://docs.djangoproject.com/en/3.1/ref/models/querysets/#...

jorl17 6y ago
I instantly thought of something similar when I looked at your code. I like the simplicity! I used to combine RabbitMQ and Celery for async tasks. Mostly because it's what I learned to do since it was already in use at my first job. But Celery is such a pain -- or, at least, it was at that job. So many configuration options, different places where they were stored in different versions[1]. Weird errors. Poor documentation (at least at the time?)... I just started going for something simpler: rq and rq-scheduler with redis. For most of my use cases it's more than enough. I've got to say that your approach has gotten me thinking of maybe simplifying everything even more. We'll see where I end up. In about 4 weeks I'll have to introduce asynchronous tasks in our current project, and though I was thinking of going the rq-way, your article has given me food for thought. Other than that, your backend stack is mostly like what we use for our projects. We also use plain old docker + docker-compose, with the small difference that we have a somewhat hacked-together system I built with bash several years ago to extend docker-compose's functionality a bit and make every component somewhat more reusable between projects and easier to fine-tune on a "per-environment" (development, staging, production, etc) basis. We also use nginx, but your article has convinced me to look into alternatives. Once again, thank you for your articles, they're a joy to read and think about! [1] To be fair, that job had several aging Django codebases and I know most of them are still stuck with Python 2 and outdated Celery and django-channels versions. I constantly kept pushing for us to get rid of technical debt, but we never got to it...it's part of the reason why me and a mate left it for our own endeavors together.
- ephimetheus 6y ago
  I’ve never been able to use rq with redis without problems. For one: restarting the app requires wiping the related redis data because rq will refuse to launch because there’s still entries for the workers, which it apparently doesn’t clean up correctly.
- simplecto 6y ago
  Interesting that you talk about gluing together some things around docker-compose. I've done something similar but on the command-line. Specifically, I use Makefile as a task-runner [1]. It is in there that I keep a lot of my custom settings for deployments, hosts, and one-off taks. [1] - https://github.com/simplecto/makefile-taskrunner-template https://github.com/simplecto/makefile-taskrunner-template
jsmeaton 6y ago
Have you looked at django-fsm for managing your state transitions? Overkill for small projects - invaluable for larger ones where state changes can be literally anywhere.
- simplecto 6y ago
  I had not seen that. Is it fair to say it is something similar to a Rules Engine like Drools?
  - jsmeaton 6y ago
    Not really. It just enforces that specific properties (status field) can only be manipulated with methods decorated by an FSM transition. Allows enforcement of what actions are taken during a state transition so that some random view doesn’t change to “in progress” without the necessary actions occurring. https://pypi.org/project/django-viewflow/ https://pypi.org/project/django-viewflow/ Is from the same developer which is a workflow/rules engine built on top of FSM.

jorl17 6y ago

I instantly thought of something similar when I looked at your code. I like the simplicity! I used to combine RabbitMQ and Celery for async tasks. Mostly because it's what I learned to do since it was already in use at my first job. But Celery is such a pain -- or, at least, it was at that job. So many configuration options, different places where they were stored in different versions[1]. Weird errors. Poor documentation (at least at the time?)... I just started going for something simpler: rq and rq-scheduler with redis. For most of my use cases it's more than enough. I've got to say that your approach has gotten me thinking of maybe simplifying everything even more. We'll see where I end up. In about 4 weeks I'll have to introduce asynchronous tasks in our current project, and though I was thinking of going the rq-way, your article has given me food for thought. Other than that, your backend stack is mostly like what we use for our projects. We also use plain old docker + docker-compose, with the small difference that we have a somewhat hacked-together system I built with bash several years ago to extend docker-compose's functionality a bit and make every component somewhat more reusable between projects and easier to fine-tune on a "per-environment" (development, staging, production, etc) basis. We also use nginx, but your article has convinced me to look into alternatives. Once again, thank you for your articles, they're a joy to read and think about! [1] To be fair, that job had several aging Django codebases and I know most of them are still stuck with Python 2 and outdated Celery and django-channels versions. I constantly kept pushing for us to get rid of technical debt, but we never got to it...it's part of the reason why me and a mate left it for our own endeavors together.

ephimetheus 6y ago
I’ve never been able to use rq with redis without problems. For one: restarting the app requires wiping the related redis data because rq will refuse to launch because there’s still entries for the workers, which it apparently doesn’t clean up correctly.
simplecto 6y ago
Interesting that you talk about gluing together some things around docker-compose. I've done something similar but on the command-line. Specifically, I use Makefile as a task-runner [1]. It is in there that I keep a lot of my custom settings for deployments, hosts, and one-off taks. [1] - https://github.com/simplecto/makefile-taskrunner-template https://github.com/simplecto/makefile-taskrunner-template

jsmeaton 6y ago

Have you looked at django-fsm for managing your state transitions? Overkill for small projects - invaluable for larger ones where state changes can be literally anywhere.

simplecto 6y ago
I had not seen that. Is it fair to say it is something similar to a Rules Engine like Drools?
- jsmeaton 6y ago
  Not really. It just enforces that specific properties (status field) can only be manipulated with methods decorated by an FSM transition. Allows enforcement of what actions are taken during a state transition so that some random view doesn’t change to “in progress” without the necessary actions occurring. https://pypi.org/project/django-viewflow/ https://pypi.org/project/django-viewflow/ Is from the same developer which is a workflow/rules engine built on top of FSM.