4 ms·

The Workflow Pattern

129 points by kiyanwang 3y ago

jusonchan81 3y ago
One of my favorite workflow engines that has a really simple way to do things was not listed here, so I'll call it out - Netflix Conductor (https://github.com/Netflix/conductor https://github.com/Netflix/conductor). Its capabilities comes to light when you model really complex workflows and one real value is how its all very visual not just during modeling but when running it. The history remains visible and you can even see how the whole flow evolved.
- pid-1 3y ago
  My company has 200+ DAGs on Airflow, so that ship has sailed for me. That said, Conductor looks really cool, thanks for sharing.
- jonnycoder 3y ago
  I recently created some Netflix Conductor workflows with Python workers and found it pretty good. Using a fork/join to kick off sub workflows was pretty easy. It definitely shows runtime variable data much better than Camunda.
cannonpalms 3y ago
I built a feature of our platform on top of Apache NiFi[1] and had a great experience. I would have preferred a non-JVM language, as that'd have been easier for us in a containerized environment, but I have no complaints about NiFi otherwise.
- claytonjy 3y ago
  Interesting; for me NiFi is tied with Airflow as my most-hated software I've had to use professionally. This was a few years ago but NiFi only supported editing the DAG from a GUI; you could store the resulting XML in git, but no human could read or edit it. Python steps were stuck on Python 2.6 or 2.7. Hard to debug pileups. Is it better now?
  - seabrookmx 3y ago
    NiFi sounds a lot like Camunda, which is awful.
jdeaton 3y ago
One of the biggest issues I've observed with workflow systems is an inability to get return values from each subprocess and then decide based on those which subprocesses to instantiate next. It always seems like the workflow execution graph is static in my experience. Is that correct?
- brad0 3y ago
  That sounds very similar to AWS Step Functions to me. You define a workflow graph in JSON, then run the workflow and hand out subprocesses to lambdas etc.
  - 8note 3y ago
    I've definitely seen workflows written as single step graphs, where the output is opening a new workflow with a different single step It was hard to read and understand
- sisve 3y ago
  Maybe before, but i feel that is not a problem anymore. Windmill.dev solves this with easee. Very good UI on top of the workflow engine.
- noselasd 3y ago
  The couple workflow engines I've used supports that. It sounds crazy to not support something that obvious
- 8note 3y ago
  Why would you want to rewrite the code of the workflow during execution, vs defining that logic upfront?
  - jdeaton 3y ago
    That's like asking why anyone would want to use an if statement. "Why would you want to change which instructions that get executed during the program? Just decide up front!". Because the decision cannot be made ahead of time.
jiggawatts 3y ago
The existence of workflow DSLs seems like a symptom of a systemic weakness in common programming languages. We use languages best suited to writing Windows GUIs or Linux kernels to implement business rules — and then we act surprised when we have to invent an entire language and runtime to solve simple business problems. One missing feature in typical languages is the native ability to have a computation frozen in the middle of a function call and then defrosted and continue as if nothing had happened. The low-level mechanisms are often available already — such as object serialisation — but these primitives never support the serialisation of a call stack or an in-flight computation. We even have compilers that can split up and restructure an async function into a heap object and an associated state machine! What’s the difference between an async function that is awaiting a slow HTTP call and an async function awaiting a long-running workflow step? Only that the state machine of the latter is persisted to storage instead of the heap! I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword. PS: The same philosophy could be applied to Java Loom style async programming where threads could be marked as eligible for hibernation, in which case they would be restricted to using data types that can be safely round-tripped by the serialiser.
- wavemode 3y ago
  The missing feature you're describing is monads. They can essentially be a first-class abstraction for computation (among other things). They can represent synchronous computations, computations that can be suspended/resumed, computations that might fail with a particular type of exception, or anything in between.
  - Karrot_Kream 3y ago
    This is one way to structure it but not the only. Scheme's call-with-current-continuation comes to mind here.
    - 3y ago
cgio 3y ago
The article is quite thorough and I enjoyed it as it reminded me of my forays in the domain. One nook in this domain, I never managed to investigate in the depth I would like is the Kinetic Rules Language [0] which makes the connection to complex event processing more pronounced. CEP is not trendy, which is probably because of history and traditional implementations but I think it deserves a good look. [0] https://github.com/Picolab/pico-engine https://github.com/Picolab/pico-engine is an implementation of the KRL
abrgr 3y ago
Modeling workflows as state machines is incredibly powerful. In addition to everything the author mentioned, the constraints of state machines allow a workflow platform to provide a ton of additional guarantees and capabilities around consistency, state propagation, reliable timers, inter-instance messaging, etc. We built our workflow execution platform [1] around state machines and we've seen great results. We find our workflow code is incredibly simple and easy to understand. [1] https://www.statebacked.dev https://www.statebacked.dev
- manvillej 3y ago
  this is exactly how I go about re-architecting overgrown workflows. reduce the workflow to a state machine. It takes forever to untangle what they are wanting to do and what they are doing, but the result is always a more robust solution. the other benefit of a statemachine is the ability to accurately determine what parts can be collapsed into subworkflows which allows for reuse, replacement, or general modification
alexisread 3y ago
One of the things I never see, is workflow integration with logging/tracing. For event-sourcing, surely the log is the best port of call? Does anyone have a pattern around this, esp wrt Clickhouse integration?

129 points by kiyanwang 3y ago

One of my favorite workflow engines that has a really simple way to do things was not listed here, so I'll call it out - Netflix Conductor (https://github.com/Netflix/conductor https://github.com/Netflix/conductor). Its capabilities comes to light when you model really complex workflows and one real value is how its all very visual not just during modeling but when running it. The history remains visible and you can even see how the whole flow evolved.

pid-1 3y ago
My company has 200+ DAGs on Airflow, so that ship has sailed for me. That said, Conductor looks really cool, thanks for sharing.
jonnycoder 3y ago
I recently created some Netflix Conductor workflows with Python workers and found it pretty good. Using a fork/join to kick off sub workflows was pretty easy. It definitely shows runtime variable data much better than Camunda.

cannonpalms 3y ago

I built a feature of our platform on top of Apache NiFi[1] and had a great experience. I would have preferred a non-JVM language, as that'd have been easier for us in a containerized environment, but I have no complaints about NiFi otherwise.

claytonjy 3y ago
Interesting; for me NiFi is tied with Airflow as my most-hated software I've had to use professionally. This was a few years ago but NiFi only supported editing the DAG from a GUI; you could store the resulting XML in git, but no human could read or edit it. Python steps were stuck on Python 2.6 or 2.7. Hard to debug pileups. Is it better now?
- seabrookmx 3y ago
  NiFi sounds a lot like Camunda, which is awful.

jdeaton 3y ago

One of the biggest issues I've observed with workflow systems is an inability to get return values from each subprocess and then decide based on those which subprocesses to instantiate next. It always seems like the workflow execution graph is static in my experience. Is that correct?

brad0 3y ago
That sounds very similar to AWS Step Functions to me. You define a workflow graph in JSON, then run the workflow and hand out subprocesses to lambdas etc.
- 8note 3y ago
  I've definitely seen workflows written as single step graphs, where the output is opening a new workflow with a different single step It was hard to read and understand
sisve 3y ago
Maybe before, but i feel that is not a problem anymore. Windmill.dev solves this with easee. Very good UI on top of the workflow engine.
noselasd 3y ago
The couple workflow engines I've used supports that. It sounds crazy to not support something that obvious
8note 3y ago
Why would you want to rewrite the code of the workflow during execution, vs defining that logic upfront?
- jdeaton 3y ago
  That's like asking why anyone would want to use an if statement. "Why would you want to change which instructions that get executed during the program? Just decide up front!". Because the decision cannot be made ahead of time.

jiggawatts 3y ago

The existence of workflow DSLs seems like a symptom of a systemic weakness in common programming languages. We use languages best suited to writing Windows GUIs or Linux kernels to implement business rules — and then we act surprised when we have to invent an entire language and runtime to solve simple business problems. One missing feature in typical languages is the native ability to have a computation frozen in the middle of a function call and then defrosted and continue as if nothing had happened. The low-level mechanisms are often available already — such as object serialisation — but these primitives never support the serialisation of a call stack or an in-flight computation. We even have compilers that can split up and restructure an async function into a heap object and an associated state machine! What’s the difference between an async function that is awaiting a slow HTTP call and an async function awaiting a long-running workflow step? Only that the state machine of the latter is persisted to storage instead of the heap! I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword. PS: The same philosophy could be applied to Java Loom style async programming where threads could be marked as eligible for hibernation, in which case they would be restricted to using data types that can be safely round-tripped by the serialiser.

wavemode 3y ago
The missing feature you're describing is monads. They can essentially be a first-class abstraction for computation (among other things). They can represent synchronous computations, computations that can be suspended/resumed, computations that might fail with a particular type of exception, or anything in between.
- Karrot_Kream 3y ago
  This is one way to structure it but not the only. Scheme's call-with-current-continuation comes to mind here.
  - 3y ago

cgio 3y ago

The article is quite thorough and I enjoyed it as it reminded me of my forays in the domain. One nook in this domain, I never managed to investigate in the depth I would like is the Kinetic Rules Language [0] which makes the connection to complex event processing more pronounced. CEP is not trendy, which is probably because of history and traditional implementations but I think it deserves a good look. [0] https://github.com/Picolab/pico-engine https://github.com/Picolab/pico-engine is an implementation of the KRL

abrgr 3y ago

Modeling workflows as state machines is incredibly powerful. In addition to everything the author mentioned, the constraints of state machines allow a workflow platform to provide a ton of additional guarantees and capabilities around consistency, state propagation, reliable timers, inter-instance messaging, etc. We built our workflow execution platform [1] around state machines and we've seen great results. We find our workflow code is incredibly simple and easy to understand. [1] https://www.statebacked.dev https://www.statebacked.dev

manvillej 3y ago
this is exactly how I go about re-architecting overgrown workflows. reduce the workflow to a state machine. It takes forever to untangle what they are wanting to do and what they are doing, but the result is always a more robust solution. the other benefit of a statemachine is the ability to accurately determine what parts can be collapsed into subworkflows which allows for reuse, replacement, or general modification

alexisread 3y ago

One of the things I never see, is workflow integration with logging/tracing. For event-sourcing, surely the log is the best port of call? Does anyone have a pattern around this, esp wrt Clickhouse integration?