4 ms·

Agree with almost everything, but Celery is pretty common in my Django projects. I don't like the complexity cost, but especially when using some PaaS for hosti

by fhd2 1y ago

Agree with almost everything, but Celery is pretty common in my Django projects. I don't like the complexity cost, but especially when using some PaaS for hosting, it's usually the least painful option. I kinda always start out thinking this time I'll manage without, and then I have a bunch of jobs triggered via HTTP calls running into timeouts. At that point it's either threads, cron jobs (tricky with PaaS) or Celery. What's your approach?

chistev 1y ago
What Paas do you use? Pythonanywhere has an always on task feature or scheduled tasks where you can run command scripts.
- fhd2 1y ago
  Heroku, AWS Elastic Beanstalk (not a fan), Fly.io. With these, it always boils down to using their Celery/RabbitMQ offering. I also used Ofelia in the past, a simple Docker image for cron jobs, but quickly outgrew it. Maybe I'm missing it, but I don't think any of these has a nice simple built-in feature.
smashed 1y ago
This. I use celery with the same code base/docker image. Just a different entry point to start a celery worker instead of a wsgi (web) worker. Too many http requests? Add web worker instances. Background jobs piling up? Add celery workers. Clearly separate read endpoints from write/transactional endpoints and you can hit a slave postgres db or the master db depending on the http call. This creates a very robust system that can scale easily from a single code base.
- physicsguy 1y ago
  I do the same, it's easy enough and doesn't require a ton of hosting logic. Out of interest, how do you run your migrations in production, deploy the service then run an ad-hoc job with the same container again? That was one thing I was never super happy with.
  - smashed 1y ago
    In an ideal world your code-base is always compatible with the previous migration state. So new version can work with the previous version's DB schema. Then, yes, simply run the migration in a transaction once the new code is deployed. Postgres has fully transactional DDL changes which helps. Of course, it heavily depends on the actual change being made. Some changes will require downtime or must be avoided if it is too heavy on the db. Another approach if the migration can be applied quickly is to just run the migrations as part of the deployment script. This will cause a downtime but can be short. Easiest is just to do runmigrations in your docker image start commands, so DB is always migrated when the container starts. tl;dr: It depends on the complexity of the migrations and your uptime requirements.
williamdclt 1y ago
You can have a queue in your relational DB (example: https://github.com/pgmq/pgmq https://github.com/pgmq/pgmq). It'll scale pretty far.
kukkeliskuu 1y ago
I was using Redis and Celery but was not happy with the complexity. But then I found out that background workers are being implemented to Django. https://github.com/django/deps/blob/main/accepted/0014-background-workers.rst https://github.com/django/deps/blob/main/accepted/0014-backg... This functionality has been backported as django-tasks library. You can use the database (i.e. Postgres) as the permanent storage. For periodic jobs and running the django-tasks, I use Superchronic, which is a Docker-compatible cron. It is compatible with cron, with the added benefit that you can run jobs with sub-minute granularity. I have had no problems with running Superchronic inside my fly.io app. I run Superchronic and Django with honcho and Procfiles.

by fhd2 1y ago

chistev 1y ago

What Paas do you use? Pythonanywhere has an always on task feature or scheduled tasks where you can run command scripts.

fhd2 1y ago
Heroku, AWS Elastic Beanstalk (not a fan), Fly.io. With these, it always boils down to using their Celery/RabbitMQ offering. I also used Ofelia in the past, a simple Docker image for cron jobs, but quickly outgrew it. Maybe I'm missing it, but I don't think any of these has a nice simple built-in feature.

smashed 1y ago

This. I use celery with the same code base/docker image. Just a different entry point to start a celery worker instead of a wsgi (web) worker. Too many http requests? Add web worker instances. Background jobs piling up? Add celery workers. Clearly separate read endpoints from write/transactional endpoints and you can hit a slave postgres db or the master db depending on the http call. This creates a very robust system that can scale easily from a single code base.

physicsguy 1y ago
I do the same, it's easy enough and doesn't require a ton of hosting logic. Out of interest, how do you run your migrations in production, deploy the service then run an ad-hoc job with the same container again? That was one thing I was never super happy with.
- smashed 1y ago
  In an ideal world your code-base is always compatible with the previous migration state. So new version can work with the previous version's DB schema. Then, yes, simply run the migration in a transaction once the new code is deployed. Postgres has fully transactional DDL changes which helps. Of course, it heavily depends on the actual change being made. Some changes will require downtime or must be avoided if it is too heavy on the db. Another approach if the migration can be applied quickly is to just run the migrations as part of the deployment script. This will cause a downtime but can be short. Easiest is just to do runmigrations in your docker image start commands, so DB is always migrated when the container starts. tl;dr: It depends on the complexity of the migrations and your uptime requirements.

williamdclt 1y ago

You can have a queue in your relational DB (example: https://github.com/pgmq/pgmq https://github.com/pgmq/pgmq). It'll scale pretty far.

kukkeliskuu 1y ago

I was using Redis and Celery but was not happy with the complexity. But then I found out that background workers are being implemented to Django. https://github.com/django/deps/blob/main/accepted/0014-background-workers.rst https://github.com/django/deps/blob/main/accepted/0014-backg... This functionality has been backported as django-tasks library. You can use the database (i.e. Postgres) as the permanent storage. For periodic jobs and running the django-tasks, I use Superchronic, which is a Docker-compatible cron. It is compatible with cron, with the added benefit that you can run jobs with sub-minute granularity. I have had no problems with running Superchronic inside my fly.io app. I run Superchronic and Django with honcho and Procfiles.