4 ms·

"Workflow engines" and their close cousins "DSLs" are the ultimate newbie trap. In theory, it's awesome for everyone: programmers get to work on interesting, ab

by mjr00 4y ago

"Workflow engines" and their close cousins "DSLs" are the ultimate newbie trap. In theory, it's awesome for everyone: programmers get to work on interesting, abstract problems like distributed systems, syntax parsing, event-based architectures, and "business users" get to make changes to "business rules" without bothering developers or impacting feature roadmaps. Win-win, right? In reality you just end up making a shitty, nerfed version of a programming language, that business users can't understand, because you still have to understand conditional logic to model workflows, oh and your documentation is terrible because devs don't bother with the boring stuff. Most of the time the devs end up implementing the workflows anyway because they don't actually work properly. If you really need a workflow engine definitely use something off-the-shelf, but I would go so far as to say that in the 95% case, you don't even need a workflow engine: you need a developer who is capable of writing some python scripts. Even if you pay a developer a full salary to do nothing but sit around and make changes to Python scripts on-demand, that's still going to be way cheaper than the complicated workflow engine solution, which will probably require a team (or multiple teams) to maintain.

__MatrixMan__ 4y ago
> the complicated workflow engine solution ... will probably require a team (or multiple teams) to maintain You can get these "as a service" which might scratch the itch for some. (Disclaimer: I work for a company that sells Airflow-as-a-service and adjacent consulting)
Simran-B 4y ago
With some tweaks to your second paragraph, the description fits CMake: In reality you just end up [with] a shitty, nerfed version of a programming language, that [...] users can't understand, because you still have to understand [CMake's] logic to [utilize its magic behavior], oh and [the] documentation is terrible because devs don't bother with the boring stuff. Most of the time [you] end up implementing [workarounds] anyway because [CMake doesn't] actually work properly.
- funnymony 4y ago
  It will work that way with anything :) Especially gradle.
- Too 4y ago
  While Cmake has a large share of idiosyncrasies, a build system is much more graph oriented and must therefor at some point have a way to declare such dependencies, often using a DSL, for concurrency and incrementals to be reliable. Lets not forget IDEs that need to understand the project structure as well. Workflows on the other hand are more conditional and imperative, mapping better to normal programming languages, with the exceptions of transient error handling, long timers and distributing workload. Here, writing a custom DSL would be a much bigger mismatch. Best for build configuration would be a hybrid, where the declaration of dependencies is done with normal programming language, which at the end calls a build(dep_tree)-function where all the magic happens. With the risk of developers abusing this setup-step and run half the build in their own reinvented imperative flow here instead... trust me, I've seen this happen even in makefiles that run shell commands outside of targets. This is what Scons tries to be, it seems however not to be very popular compared to cmake.
  - Simran-B 4y ago
    Isn't Scons abandon-ware? I remember it being part of the build setup at work at some point and it caused quite a bit of frustrations because it broke every now and then. I believe it was mainly failing on Windows. Conan's recent development is promising and it gives you the full power of Python. It can also be used declaratively but with limitations. If I remember correctly, there are discussions in the C++ world about a declarative exchange format for dependencies and build information, but it's in the early stages. It's not trivial because there are also C++ modules now.
pak9rabid 4y ago
Damn, this hits the nail right on the head.
rwmj 4y ago
This reminded me of the time I wrote a mini-language for querying web server log files and generating reports (this was early 2000s) so you could answer questions like "how many users searched for 'cyprus flight' per day and went on to purchase a holiday?". With a nice web-based interface etc. It was a disaster of course. The business people weren't programmers, had no understanding of programming, didn't want to learn programming even greatly simplified with docs and examples, and I ended up translating their queries into code anyway. So essentially I had written a DSL for myself.
lorendsr 4y ago
I agree about the limitations of DSLs. Temporal is a workflow engine that doesn't use a DSL or nerfed version of a programming language. It runs your arbitrary code with any of the supported runtimes (currently Go/Java/Node/Python). I'm able to write, deploy, and maintain workflow code by myself and use their cloud service for persistence and admin UI.
bbbobbb 4y ago
You're probably thinking of a different type of "workflow engine" than what Temporal is. It's not something where business people drag boxes in a GUI. It's still all code, within your code base, only with different approach to handling long running (whatever that means in your context - 2 minutes or 2 years) tasks.
eternityforest 4y ago
Low level infrastructure of any kind is the worst. Programmers love to work on it, so there's never a shortage of good implementations, companies need it, so some of them get proper funding and teams, and yet it's really hard so there's no way your DIY solution will have the features of the big ones, and it probably won't have the reliability. And worse, you'll probably need to learn and support the existing thing anyway, why not just skip to it?

by mjr00 4y ago

__MatrixMan__ 4y ago

> the complicated workflow engine solution ... will probably require a team (or multiple teams) to maintain You can get these "as a service" which might scratch the itch for some. (Disclaimer: I work for a company that sells Airflow-as-a-service and adjacent consulting)

Simran-B 4y ago

With some tweaks to your second paragraph, the description fits CMake: In reality you just end up [with] a shitty, nerfed version of a programming language, that [...] users can't understand, because you still have to understand [CMake's] logic to [utilize its magic behavior], oh and [the] documentation is terrible because devs don't bother with the boring stuff. Most of the time [you] end up implementing [workarounds] anyway because [CMake doesn't] actually work properly.

funnymony 4y ago
It will work that way with anything :) Especially gradle.
Too 4y ago
While Cmake has a large share of idiosyncrasies, a build system is much more graph oriented and must therefor at some point have a way to declare such dependencies, often using a DSL, for concurrency and incrementals to be reliable. Lets not forget IDEs that need to understand the project structure as well. Workflows on the other hand are more conditional and imperative, mapping better to normal programming languages, with the exceptions of transient error handling, long timers and distributing workload. Here, writing a custom DSL would be a much bigger mismatch. Best for build configuration would be a hybrid, where the declaration of dependencies is done with normal programming language, which at the end calls a build(dep_tree)-function where all the magic happens. With the risk of developers abusing this setup-step and run half the build in their own reinvented imperative flow here instead... trust me, I've seen this happen even in makefiles that run shell commands outside of targets. This is what Scons tries to be, it seems however not to be very popular compared to cmake.
- Simran-B 4y ago
  Isn't Scons abandon-ware? I remember it being part of the build setup at work at some point and it caused quite a bit of frustrations because it broke every now and then. I believe it was mainly failing on Windows. Conan's recent development is promising and it gives you the full power of Python. It can also be used declaratively but with limitations. If I remember correctly, there are discussions in the C++ world about a declarative exchange format for dependencies and build information, but it's in the early stages. It's not trivial because there are also C++ modules now.

pak9rabid 4y ago

Damn, this hits the nail right on the head.

rwmj 4y ago

This reminded me of the time I wrote a mini-language for querying web server log files and generating reports (this was early 2000s) so you could answer questions like "how many users searched for 'cyprus flight' per day and went on to purchase a holiday?". With a nice web-based interface etc. It was a disaster of course. The business people weren't programmers, had no understanding of programming, didn't want to learn programming even greatly simplified with docs and examples, and I ended up translating their queries into code anyway. So essentially I had written a DSL for myself.

lorendsr 4y ago

I agree about the limitations of DSLs. Temporal is a workflow engine that doesn't use a DSL or nerfed version of a programming language. It runs your arbitrary code with any of the supported runtimes (currently Go/Java/Node/Python). I'm able to write, deploy, and maintain workflow code by myself and use their cloud service for persistence and admin UI.

bbbobbb 4y ago

You're probably thinking of a different type of "workflow engine" than what Temporal is. It's not something where business people drag boxes in a GUI. It's still all code, within your code base, only with different approach to handling long running (whatever that means in your context - 2 minutes or 2 years) tasks.

eternityforest 4y ago

Low level infrastructure of any kind is the worst. Programmers love to work on it, so there's never a shortage of good implementations, companies need it, so some of them get proper funding and teams, and yet it's really hard so there's no way your DIY solution will have the features of the big ones, and it probably won't have the reliability. And worse, you'll probably need to learn and support the existing thing anyway, why not just skip to it?