4 ms·

The existence of workflow DSLs seems like a symptom of a systemic weakness in common programming languages. We use languages best suited to writing Windows GUI

The existence of workflow DSLs seems like a symptom of a systemic weakness in common programming languages. We use languages best suited to writing Windows GUIs or Linux kernels to implement business rules — and then we act surprised when we have to invent an entire language and runtime to solve simple business problems. One missing feature in typical languages is the native ability to have a computation frozen in the middle of a function call and then defrosted and continue as if nothing had happened. The low-level mechanisms are often available already — such as object serialisation — but these primitives never support the serialisation of a call stack or an in-flight computation. We even have compilers that can split up and restructure an async function into a heap object and an associated state machine! What’s the difference between an async function that is awaiting a slow HTTP call and an async function awaiting a long-running workflow step? Only that the state machine of the latter is persisted to storage instead of the heap! I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword. PS: The same philosophy could be applied to Java Loom style async programming where threads could be marked as eligible for hibernation, in which case they would be restricted to using data types that can be safely round-tripped by the serialiser.

wavemode 3y ago
The missing feature you're describing is monads. They can essentially be a first-class abstraction for computation (among other things). They can represent synchronous computations, computations that can be suspended/resumed, computations that might fail with a particular type of exception, or anything in between.
- Karrot_Kream 3y ago
  This is one way to structure it but not the only. Scheme's call-with-current-continuation comes to mind here.
  - code_biologist 3y ago
    One really nifty use case of continuations is the Seaside Smalltalk web framework [1]. It implements stateful request/response conversations with Smalltalk's ability to capture and serialize continuations. Continuation-based web frameworks are really nifty, but have potential scaling bottlenecks. IIRC HN originally used some type of continuation web framework with Arc, but ran into scaling and caching headaches. Dunno how it works these days. [1] https://en.wikipedia.org/wiki/Seaside_(software) https://en.wikipedia.org/wiki/Seaside_(software)
    - codesnik 3y ago
      I didn't know it can serialize continuation! I always thought that with seaside you have to run a single server, and any reboot would just kill the state.
  - alexisread 3y ago
    I didn't think callCC can do multiple resumptions? DDBinding certainly can, and can do partial environment closures so you can migrate a process to a different machine as well: https://okmij.org/ftp/papers/DDBinding.pdf https://okmij.org/ftp/papers/DDBinding.pdf The above is succinctly implemented in wat: https://github.com/manuel/wat-js https://github.com/manuel/wat-js (also see forks for more documented versions).
vb-8448 3y ago
Workflows engines are not just about deferring a http call or a specific language. They are a generic way to run some background process that doesn't need too much human interaction and can be coded in any language. If you need to orchestrate tens of thousands of tasks, you will beg for a DSL and some kind of interface to monitor things.
- manvillej 3y ago
  I feel like awaiting some human interaction is one of my most common requests for workflows. await some approval or some manual task that can't be automated, etc. honestly, when working with some new archaic system, the ability to insert manual tasks is invaluable for an early deployment
dartos 3y ago
The async/await stuff in JavaScript, at least, is supported by the “generators” feature, which does let you effectively “pause” execution of a function, retaining its state. You can then yield to that state to continue executing
- vanviegen 3y ago
  Sure, but would you still be able to resume that function 2 weeks later, after 5 process restarts due to code deploys and on a different server?
  - Izkata 3y ago
    CRIU can apparently do that with whole processes: https://criu.org/Main_Page https://criu.org/Main_Page Found a link to it from telefork: https://thume.ca/2020/04/18/telefork-forking-a-process-onto-a-different-computer/ https://thume.ca/2020/04/18/telefork-forking-a-process-onto-...
kodablah 3y ago
> I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword. This is how Temporal works. For example, in Python the async event loop is replaced by a durable event loop [0], the JS promises become durable promises, .NET tasks become durable via a custom scheduler, etc. Granted it doesn't serialize the stack, it uses event sourcing almost exactly like the article describes, and therefore requires deterministic code for replaying. From the dev POV, it looks like any code can just be frozen in the middle of a function and magically resumed elsewhere. 0 - https://temporal.io/blog/durable-distributed-asyncio-event-loop https://temporal.io/blog/durable-distributed-asyncio-event-l... (disclaimer, I work at Temporal and have written some of these distributed coroutine impls)
mattnewport 3y ago
Sounds like Azure Durable Functions: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=in-process%2Cnodejs-v3%2Cv1-model&pivots=csharp https://learn.microsoft.com/en-us/azure/azure-functions/dura...
svilen_dobrev 3y ago
and there is always the Long-Running-Transactions. Where the workflow/state-machine can take days or weeks to finish. Somehow, we all still live with some hope that certain things will happen real fast so they can live in threads/processes/memory (and have no persistent backing, except maybe retry whole thing), while those that can take "longer" (how long is "longer"?) should be completely different beasts. While from business-level view, there is no much difference - but they are programmed in completely different way. For example, in some businesses, payment is like another communication. Not that sending a single message across is very simple thing, but definitely a lot less complicated than a payment process. Maybe the split, should it be internaly-language-handled - and forgotten if power goes off - or externalized and persisted, should depend on where there is certain boundary being crossed? Defining "boundary" per project..
jauntywundrkind 3y ago
Rather than focus on the DSL/code-writing aspect of workflow pattern tools, I feel the crucial difference is that most workflow systems are intended for a broad audience of users, many of whom are not devs. I have similar doubts about how important the difference is between programming as devs do it and these tools. But to me, it's less a language concern and more a question of how tasks/jobs/processes/whatever-you-call-them are managed, if they're managed at all. In programming languages today, devs spawn countless async sub-processes (promises, async tasks, whatever you call them) with abandon. And they're implicit and invisible in almost all systems. An operator or user isn't there introspecting what jobs are outstanding. Or managing the workflow of these subprocesses. These workflow tools are just a tiny leap over what programmers do: they make computing real. They reify computing into a managed element, make work being done a kind of data that is tracked through the system. Where-as in most programming, we have lots of tools for authoring and finagling subprocesses, but we are so very lacking in tools to manage & expose & let the user control what's happening in the process. The processs is mostly-closed, except what is afforded, and what is afforded is largely custom-craft APIs (with folks like OTel starting to define some more common ways for processes to at least explain themselves, if not manage & interconnect with other processes). I don't think the languages have to change. I think the languages can stay right where they are, for the most part. But we can and should be building better runtimes, that let us interact with the objects & subprocesses, that let the app be more commanded and turned into an task runner rather than also the command-pallet system, kind of has to happen. It's not super related, but there was a post on Audacity in the browser, where @solardev suggested "headless" WASM apps and then separate UIs, and I can easily see that idea applying here: a workflow pattern tool being the UI, that orchestrates/drives WASM apps (which are practically just libraries).

wavemode 3y ago

The missing feature you're describing is monads. They can essentially be a first-class abstraction for computation (among other things). They can represent synchronous computations, computations that can be suspended/resumed, computations that might fail with a particular type of exception, or anything in between.

Karrot_Kream 3y ago
This is one way to structure it but not the only. Scheme's call-with-current-continuation comes to mind here.
- code_biologist 3y ago
  One really nifty use case of continuations is the Seaside Smalltalk web framework [1]. It implements stateful request/response conversations with Smalltalk's ability to capture and serialize continuations. Continuation-based web frameworks are really nifty, but have potential scaling bottlenecks. IIRC HN originally used some type of continuation web framework with Arc, but ran into scaling and caching headaches. Dunno how it works these days. [1] https://en.wikipedia.org/wiki/Seaside_(software) https://en.wikipedia.org/wiki/Seaside_(software)
  - codesnik 3y ago
    I didn't know it can serialize continuation! I always thought that with seaside you have to run a single server, and any reboot would just kill the state.
- alexisread 3y ago
  I didn't think callCC can do multiple resumptions? DDBinding certainly can, and can do partial environment closures so you can migrate a process to a different machine as well: https://okmij.org/ftp/papers/DDBinding.pdf https://okmij.org/ftp/papers/DDBinding.pdf The above is succinctly implemented in wat: https://github.com/manuel/wat-js https://github.com/manuel/wat-js (also see forks for more documented versions).

vb-8448 3y ago

Workflows engines are not just about deferring a http call or a specific language. They are a generic way to run some background process that doesn't need too much human interaction and can be coded in any language. If you need to orchestrate tens of thousands of tasks, you will beg for a DSL and some kind of interface to monitor things.

manvillej 3y ago
I feel like awaiting some human interaction is one of my most common requests for workflows. await some approval or some manual task that can't be automated, etc. honestly, when working with some new archaic system, the ability to insert manual tasks is invaluable for an early deployment

dartos 3y ago

The async/await stuff in JavaScript, at least, is supported by the “generators” feature, which does let you effectively “pause” execution of a function, retaining its state. You can then yield to that state to continue executing

vanviegen 3y ago
Sure, but would you still be able to resume that function 2 weeks later, after 5 process restarts due to code deploys and on a different server?
- Izkata 3y ago
  CRIU can apparently do that with whole processes: https://criu.org/Main_Page https://criu.org/Main_Page Found a link to it from telefork: https://thume.ca/2020/04/18/telefork-forking-a-process-onto-a-different-computer/ https://thume.ca/2020/04/18/telefork-forking-a-process-onto-...

kodablah 3y ago

> I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword. This is how Temporal works. For example, in Python the async event loop is replaced by a durable event loop [0], the JS promises become durable promises, .NET tasks become durable via a custom scheduler, etc. Granted it doesn't serialize the stack, it uses event sourcing almost exactly like the article describes, and therefore requires deterministic code for replaying. From the dev POV, it looks like any code can just be frozen in the middle of a function and magically resumed elsewhere. 0 - https://temporal.io/blog/durable-distributed-asyncio-event-loop https://temporal.io/blog/durable-distributed-asyncio-event-l... (disclaimer, I work at Temporal and have written some of these distributed coroutine impls)

mattnewport 3y ago

Sounds like Azure Durable Functions: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=in-process%2Cnodejs-v3%2Cv1-model&pivots=csharp https://learn.microsoft.com/en-us/azure/azure-functions/dura...

svilen_dobrev 3y ago

and there is always the Long-Running-Transactions. Where the workflow/state-machine can take days or weeks to finish. Somehow, we all still live with some hope that certain things will happen real fast so they can live in threads/processes/memory (and have no persistent backing, except maybe retry whole thing), while those that can take "longer" (how long is "longer"?) should be completely different beasts. While from business-level view, there is no much difference - but they are programmed in completely different way. For example, in some businesses, payment is like another communication. Not that sending a single message across is very simple thing, but definitely a lot less complicated than a payment process. Maybe the split, should it be internaly-language-handled - and forgotten if power goes off - or externalized and persisted, should depend on where there is certain boundary being crossed? Defining "boundary" per project..

jauntywundrkind 3y ago

Rather than focus on the DSL/code-writing aspect of workflow pattern tools, I feel the crucial difference is that most workflow systems are intended for a broad audience of users, many of whom are not devs. I have similar doubts about how important the difference is between programming as devs do it and these tools. But to me, it's less a language concern and more a question of how tasks/jobs/processes/whatever-you-call-them are managed, if they're managed at all. In programming languages today, devs spawn countless async sub-processes (promises, async tasks, whatever you call them) with abandon. And they're implicit and invisible in almost all systems. An operator or user isn't there introspecting what jobs are outstanding. Or managing the workflow of these subprocesses. These workflow tools are just a tiny leap over what programmers do: they make computing real. They reify computing into a managed element, make work being done a kind of data that is tracked through the system. Where-as in most programming, we have lots of tools for authoring and finagling subprocesses, but we are so very lacking in tools to manage & expose & let the user control what's happening in the process. The processs is mostly-closed, except what is afforded, and what is afforded is largely custom-craft APIs (with folks like OTel starting to define some more common ways for processes to at least explain themselves, if not manage & interconnect with other processes). I don't think the languages have to change. I think the languages can stay right where they are, for the most part. But we can and should be building better runtimes, that let us interact with the objects & subprocesses, that let the app be more commanded and turned into an task runner rather than also the command-pallet system, kind of has to happen. It's not super related, but there was a post on Audacity in the browser, where @solardev suggested "headless" WASM apps and then separate UIs, and I can easily see that idea applying here: a workflow pattern tool being the UI, that orchestrates/drives WASM apps (which are practically just libraries).