Have you looked at AWS Step Functions? https://aws.amazon.com/step-functions https://aws.amazon.com/step-functions edit: to add, I would highly recommend using

Have you looked at AWS Step Functions? https://aws.amazon.com/step-functions https://aws.amazon.com/step-functions edit: to add, I would highly recommend using a workflow engine over a distributed messaging system. With messages it's hard to track where a given work item is in your pipeline, and it's not always easy to do mass operations such as just stopping all running workflows (e.g. when you have an outage) and resuming them later, re-driving failed items from the beginning of the workflow, etc. Workflow engines typically give you a nice dashboard where you can do all those things, for free.