3 ms·

Can someone explain how and where to use a Workflow Orchestration Engine ?

juancampa 6y ago
These are useful for tasks that can last an arbitrarily long amount of time. Think about the process of signing up a user and waiting for her to click on a email verification link. This process can literally never end (the user never clicked) but more commonly it takes a few minutes. It's easier to implement these things if you can write the code like: await sendVerificationEmail(); await waitForUserToClick(); // This could take forever await sendWelcomeEmail(); If you do the above in a "normal" program, said program could stay in memory forever (consuming RAM, server can't restart, etc). The workflow engine will take care of storing intermediate state so you can indeed write the above code. The other option is to implement a state machine via your database and some state column, but the code doesn't look as pretty as the above three lines. Note that this particular tool seems to be more declarative than my example above (it uses JSON to do define the steps), so instead of using an `if` statement, you'd need to declare a "Decision". Hope this helps!
- vvladymyrov 6y ago
  So is it like https://github.com/uber/cadence https://github.com/uber/cadence, AWS SWF, Amazon Step Functions?
  - ecnahc515 6y ago
    Yes, they all solve similar problems. This is a space with a lot of differing requirements, so they all share some common features, but many have different strengths for various types of tasks.
- bogomipz 6y ago
  Interesting. So what holds the state in a workflow orchestrator? In your example what if user never clicks? Or if a server is rebooted? Is the state persisted to disk and then garbage collected after after a certain threshold?
  - juancampa 6y ago
    > Is the state persisted to disk Exactly!
    - swyx 6y ago
      other benefit of this is the process also becomes "fault oblivious" - any part of it can crash and restart and it picks up where it left off
- acjohnson55 6y ago
  Yep. This is the dream, IMO. Serverless functions that can await for arbitrary spans of time. It's incredible to me how much effort we spend working around this limitation. Cadence/Temporal are pretty close to this ideal, but I need to look into what Conductor has going on.
stingraycharles 6y ago
I’ve seen them a lot, but two of the more canonical use cases are (complex) ETL and Continuous Delivery processes. In both these cases, before these types of systems existed, it were useful a dozens of ad-hoc scripts that handled it. Monitoring, testing, instrumentation and visibility were usually after-thoughts and often not worth the trouble. Workflow Orchestration Engines solve these problems. Each of these workflow engines have a slightly different angle, slightly better at different use cases, and as usual it can be quite important to select the right tool for the job. But if you have found one, it can be a real boost to the quality of your processes.

These are useful for tasks that can last an arbitrarily long amount of time. Think about the process of signing up a user and waiting for her to click on a email verification link. This process can literally never end (the user never clicked) but more commonly it takes a few minutes. It's easier to implement these things if you can write the code like: await sendVerificationEmail(); await waitForUserToClick(); // This could take forever await sendWelcomeEmail(); If you do the above in a "normal" program, said program could stay in memory forever (consuming RAM, server can't restart, etc). The workflow engine will take care of storing intermediate state so you can indeed write the above code. The other option is to implement a state machine via your database and some state column, but the code doesn't look as pretty as the above three lines. Note that this particular tool seems to be more declarative than my example above (it uses JSON to do define the steps), so instead of using an `if` statement, you'd need to declare a "Decision". Hope this helps!

vvladymyrov 6y ago
So is it like https://github.com/uber/cadence https://github.com/uber/cadence, AWS SWF, Amazon Step Functions?
- ecnahc515 6y ago
  Yes, they all solve similar problems. This is a space with a lot of differing requirements, so they all share some common features, but many have different strengths for various types of tasks.
bogomipz 6y ago
Interesting. So what holds the state in a workflow orchestrator? In your example what if user never clicks? Or if a server is rebooted? Is the state persisted to disk and then garbage collected after after a certain threshold?
- juancampa 6y ago
  > Is the state persisted to disk Exactly!
  - swyx 6y ago
    other benefit of this is the process also becomes "fault oblivious" - any part of it can crash and restart and it picks up where it left off
acjohnson55 6y ago
Yep. This is the dream, IMO. Serverless functions that can await for arbitrary spans of time. It's incredible to me how much effort we spend working around this limitation. Cadence/Temporal are pretty close to this ideal, but I need to look into what Conductor has going on.

stingraycharles 6y ago

I’ve seen them a lot, but two of the more canonical use cases are (complex) ETL and Continuous Delivery processes. In both these cases, before these types of systems existed, it were useful a dozens of ad-hoc scripts that handled it. Monitoring, testing, instrumentation and visibility were usually after-thoughts and often not worth the trouble. Workflow Orchestration Engines solve these problems. Each of these workflow engines have a slightly different angle, slightly better at different use cases, and as usual it can be quite important to select the right tool for the job. But if you have found one, it can be a real boost to the quality of your processes.