* Orchestrated, in-house built workflow execution engine. * Message queue based with service listeners that translate and dispatch messages to individual worke

* Orchestrated, in-house built workflow execution engine. * Message queue based with service listeners that translate and dispatch messages to individual workers via HTTP requests. * Workflow execution state is currently backed by RDBMS. * Infrastructure errors with workflow executions are exceedingly rare and devops can push a button to retry a step if they failed due to a transient condition. Now, the important bit: * Retries due to business logic error aren't really a thing unless there is a defined recovery transition for that step in the workflow. This forces people to acknowledge their code did something unexpected (or the workflow definition itself doesn't properly handle all necessary error cases) and fix the underlying issue. Once the root cause is identified and fixed, the workflow instance can be canceled and resubmitted. However, since things that do work usually have side effects, there is sometimes manual cleanup that has to occur that falls on the development team to fix (with assistance from the devops team, if needed). No one likes doing cleanups or getting on devops bad side so there is an incentive to make sure code and workflows are well tested before being released to production.