4 ms·
Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve t
by Sparkyte 2y ago
Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve this but most cloud providers have queuing tools.
- sjansen 2y agoPutting work in a queue is only the start. Most organizations start there and gradually write ad hoc logic as they discover problems like dependencies, retries, & scheduling. Dependencies: what can be done in parallel and what must be done in sequence? For example, three tasks get pushed in the queue and only after all three finish a fourth task must be run. Retries: The concept is simple. The details are killer. For example, ifa task fails, how long should the delay between retries be? Too short and you create a retry storm. Forget to add some jitter and you get thundering hoards all retrying at the same time. Scheduling: Because cron is good enough, until it isn't. A good workflow solution provides battle tested versions of all of the above. Better yet, a great workflow solution makes it easier to keep business logic separate from plumbing so that it's easier to reason about and test.
- shawabawa3 2y agoworkflows typically involve chains of jobs with state transitions, waits, triggers, error handling etc a lot more than just e.g. celery jobs
- nijave 2y agoA workflow manager implements a Choreography based saga pattern https://microservices.io/patterns/data/saga.html https://microservices.io/patterns/data/saga.html