3 ms·
There are great comments on the technology itself, including SQS/PubSub/RabbitMQ, or Celery. However if you need 1) management, i.e. easy way to look at what w
by ecesena 8y ago
There are great comments on the technology itself, including SQS/PubSub/RabbitMQ, or Celery.
However if you need 1) management, i.e. easy way to look at what went wrong and retry 2) dependencies between jobs, you should look into something like Airflow (instead of building your own): https://airflow.apache.org https://airflow.apache.org
It's also a very good example of architecture, in case you decide it's not good for you and you really want to build your own.