I work on argo. The workflow-controller is very tolerant to crashes and designed to be this way. Workflow state is captured in the workflow CRD object (in k8s e

I work on argo. The workflow-controller is very tolerant to crashes and designed to be this way. Workflow state is captured in the workflow CRD object (in k8s etcd). Because step names are formulated, in the event of a crash (say before the created pod is persisted in etcd), when the controller restarts and tries to schedule the pod again, it hits an AlreadyExists error and understands how to handle this. Thus, workflows are idempotent in crash scenarios.