10 ms·

Cadence: Uber's Workflow Orchestration Engine

243 points by vruiz 7y ago

fokinsean 7y ago
Sounds cool, but even after poking through the repos I still don't fully understand what it does.
- maxmcd 7y ago
  The linked talk provides context and use cases within the first 10 minutes or so: https://atscaleconference.com/videos/cadence-microservice-architecture-beyond-requestreply/ https://atscaleconference.com/videos/cadence-microservice-ar...
dlphn___xyz 7y ago
how does this compare to Luigi or Airflow?
- daveFNbuck 7y ago
  Cadence says it's meant for long-running tasks, while Luigi tasks are supposed to be short.
  - mfateev 7y ago
    Cadence supports both long-runnig as well as very short tasks.
    - daveFNbuck 7y ago
      I didn't think that Cadence would fail if the task finished too quickly. The point is that Cadence is designed to solve a slightly different problem than Luigi.
- mfateev 7y ago
  The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence. I'm from the Cadence team.
  - davis_m 7y ago
    > Obviously it is also a good fit for low scale use cases. Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.
    - mfateev 7y ago
      We have both very low scale use cases as a single distributed cron as well as very high scale use cases in production at Uber.
  - zimablue 7y ago
    As someone who uses (but doesn't love, actually) Airflow. I think this might be a situation where lots of people are tempted to think that they're "scale". I disagree slightly with your use of "business level" to describe this. I've seen quant hedge fund platforms(fairly intensive data engines compared to most businesses) using airflow, because when it comes down to it hundreds of DAGs is more likely than hundreds of thousands.
- alfalfasprout 7y ago
  I would strongly suggest not using Airflow if your company doesn't already... and that's coming from someone working at the company that made it. 1) It has no knowledge of data dependencies. This means task success is very primitive and you end up with a bunch of "polling" tasks waiting eg; for a table partition to land. 2) The UI constantly hangs and/or crashes 3) Airflow "workers" using Celery are rarely correctly given the right numbers of tasks. OOM-ing, etc. are all commonplace even if using Docker. And many, many more. I strongly suggest using Apache Beam or Argo w/ Kubernetes instead. They can scale quite a bit more and deal with long running tasks well.
  - red_hare 7y ago
    I'm actually finding airflow + beam + kubernetes to be a really powerful combination. Especially if you're on GCP where airflow is managed by Cloud Composer, kubernetes is managed by GKE, and beam is managed by Dataflow. Our airflow cluster _does_ almost nothing. Our operators just tell kubernetes to run containers with commands or dataflow to run some beam template and waits for the results from afar. I love this setup but man do I agree with your points about the UI. For a product so powerful I can't believe how problematic it is.
formalsystem 7y ago
Does anyone know what are the best OSS Orchestration Engines? Am wondering what I should be comparing this to.
- manyxcxi 7y ago
  I don’t know about best, as there are lots of trade offs between various projects. I’ve reviewed and worked with a lot of Java based ones, top of mind: - Airflow - jBPM - Enhydra Shark - Activiti - Netflix Conductor Most of those are more workflow engine than pure orchestration. A few years back, we searched high and low and I was generally unhappy with the commercial offerings so we wound up building around Netflix Conductor (after a disastrous run with Joget, which is built on Enhydra Shark). Since then I’ve been pretty happy with Conductor, submitted numerous PRs and accidentally became one of the people keeping the MySQL backend implementation going forward.
  - mfateev 7y ago
    AFAIK Netflix Conductor was inspired by the AWS Simple Workflow engine. The Cadence is a direct evolution of SWF in the open source world. I was tech lead for both of them :).
  - opportune 7y ago
    I guess technically Oozie is also a FOSS Java-based workflow/orchestration engine, except it's mostly for Hadoop jobs. Haven't used too many orchestration engines but I found Oozie very barebones
  - zok3102 7y ago
    You may also want to look at Project Flogo (flogo.io) - in particular, the Flogo Flow action. It's a process engine in Golang that can be used to implement service orchestration, event-driven choreography integration patterns. Flogo has been focused more on stateless service choreography running in Serverless FaaS or Edge devices - however the underlying process engine can be used to implement a long running orchestration process with fully externalized state for resilience and scale. It's 3-clause BSD including the modeling UI with hundreds of triggers and activity extensions in the community. We have a fairly active gitter channel in case you have questions for the core team (https://gitter.im/project-flogo/Lobby https://gitter.im/project-flogo/Lobby) Disclaimer: I work at TIBCO & we provide commercial support for Project Flogo as well as use it in our commercial products.
redact207 7y ago
I've been working on a similar workflow engine for node at https://www.npmjs.com/package/@node-ts/bus-workflow https://www.npmjs.com/package/@node-ts/bus-workflow The main objective of workflows is to manage long running processes. By processes I mean business processes like coordinating the activities of fulfilling a customer order (settling charges, picking inventory, packing, dispatching, email receipts etc). It's a way to keep all those individual commands decoupled but coordinate them at a higher level. This isn't a new concept by any means, and is often paired with Domain Driven Design and message based systems. Doing so gives you a library of events everytime something happens in your system that can be reacted to in a workflow. If you've ever dealt with microservices, or even a monolith where two internal services are incorrectly coupled together then this approach may be worth looking into.
- wetpaste 7y ago
  Thank you. I feel kind of silly about this but I feel like I've had a hard time understanding when an org should, or could use something like this. I have seen them mentioned but every time it's explained it's explained with more abstract language on top of it that confuses me. I keep hearing "it manages business processes" but then it fails to mention if this means like, a human being's process within an org, or something coupled with an application of some sort that has business processes in the application? Does this type of thing replace sort of what Jira does, make a ticket and then pass it off to the next team or whatever? Do you ship it with the app for on-premise deployments of a software product? I have a hard time seeing the big picture with things like this sometimes. Then I hear workflow orchestrator and I think, oh okay so like ansible, but for, work...flows? But what is a workflow really exactly?
  - mfateev 7y ago
    My personal opinion is that the workflow is any business logic that goes beyond a single request reply. Examples of workflows: * Service deployment. * Uber trip * Media processing (download file, transcode, upload result) * Order processing * Customer support ticket processing * Customer incentive program management * Data pipeline processing * ML Training * Distributed CRON * Customer signup flow and many others.
ajbosco 7y ago
I thought they used Piper (based on Airflow) for workflows at Uber. https://eng.uber.com/managing-data-workflows-at-scale/ https://eng.uber.com/managing-data-workflows-at-scale/
- thor24 7y ago
  That is for data workflows (your etl jobs basically). This is for services.
  - dgladkov 7y ago
    If you already have a service that requires ETL, Cadence might be a good choice as well. You will need more boilerplate and setup compared to Airflow/Piper, but you can share data structures between ETLs and services that use it and have more control over execution and deployment as you own your workers.
vxa_victor 7y ago
How does it compare to BPM solution like Camunda?
- mfateev 7y ago
  Cadence allows to write workflows as code. Think of it as a virtual machine for OO code that makes that code fully fault tolerant to process failures. As it is code it can be used to implement any business logic. Camunda is BPMN engine which interprets BPMN workflow definition. It is possible to implement a Cadence workflow that interprets BPMN without changing the core Cadence service.
  - _ph_ 7y ago
    And we are talking about a software company and a software package. Sounds like a strong overlap to me - or do you think you can call a new software library "microsoft"?
bozoUser 7y ago
Looking at a few comments, can someone answer on whats are the nuances between managing data workflows vs service workflows ?
mshockwave 7y ago
just a random comment: I thought Cadence, the EDA company, owns the trademark for the name
- jacques_chester 7y ago
  Trademarks are partitioned by subject matter: https://www.uspto.gov/web/patents/classification/selectnumwithtitle.htm https://www.uspto.gov/web/patents/classification/selectnumwi...
iblaine 7y ago
Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours. Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.
- mfateev 7y ago
  Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.
  - iblaine 7y ago
    Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?
    - mfateev 7y ago
      For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.
      - chrischen 7y ago
        For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events? Also how does it listen for events? From polling?
        mfateev 7y ago
deleted 7y ago
[deleted]
jbbarth 7y ago
Cadence looks like an OSS version of Amazon Simple Workflow (SWF) service. The author used to work on SWF at AWS afaik. I'm a heavy SWF user at work for managing complex data pipelines. SWF requires an important conceptual and tooling effort in the beginning, but it gets reimbursed if you use it a lot. As for other comments mentioning Airflow: the programming model is quite different , since Airflow as far as I understand forces to provide a DAG of tasks upfront. SWF (and Cadence?) doesn't, it coordinates the work of Deciders and Activity Workers and only acts as a source of truth for the state of the workflow (+ distribute task in a unique manner to many long-polling workers). As a result you don't declare anything upfront and can have deciders take dynamic decisions along the way, which is really nice when you want very dynamic logic for your workflows (e.g. dynamic partitioning of tasks, decisions depending on external factors, etc.). I'd love to have Maxim insights about how Cadence compares to SWF, and what would be the reasons/challenges behind migrating from SWF to Cadence for SWF users (except that SWF is basically stale for 4+ years and rigged with arbitrary limits)
- mfateev 7y ago
  Cadence vs SWF Cadence was conceived and is still led by the original tech leads of the SWF. SWF had no new features added for the last 5 years. Cadence is open sourced and is under active development. Cadence was initially based on SWF public API. It uses Thrift and TChannel for communication and SWF uses AWS version of REST. Currently the API is not compatible with SWF as Cadence added a large number of new features and deprecated a few problematic ones. We are planning migrating to gRPC later this year. Cadence can potentially run on any database that supports single shard multi-row transactions as a backend. Currently it supports Cassandra and MySQL. SWF has pretty tight throttling limits. Cadence scales very well with use cases in production that require 100s of millions of open workflows and thousands of events per second. SWF has pretty tight limits on individual payloads and number of events. For example maximum activity input size is 32k. Cadence currently has 256k limit. SWF history size limit is 10k events while Cadence limit 200k. All other limits are also higher. Cadence has no limit on the activity and workflow execution duration. Cadence through archival supports unlimited retention after a workflow closure. SWF has Java and Ruby client libraries. Cadence has Java and Go client libraries. SWF Java library is fully asynchronous and relies on both code generation (through annotation processor) and AspectJ. It is hard to set up, doesn't play well with IDEs and has very steep learning curve. Cadence Java library (as well as Go one) allow writing workflows as synchronous programs which greatly simplifies the programming model. It also just a library without any need for code generation or AspectJ or similar intrusive technologies. Cadence client side libraries have much better unit testing support. For example the Java library utilizes an in-memory implementation of the Cadence service. Cadence features that SWF doesn't have: Workflow stickiness. SWF replays the whole workflow history on every decision. Which means that a workflow resource usage is proportional to O(n*n) of number of events in the history. Cadence caches workflows on a worker and delivers only new events to them. The whole history is replayed only when a worker goes down or the workflow gets out of cache. So Cadence workflow resource usage is O(n) of number of events in the history. For large workflows it makes a huge difference. It also leads to higher per workflow scale. For example it is not recommended to have workflows that execute over a hundred activities in SWF. Cadence routinely executes workflows that have over thousand activities or child workflows. Query workflow execution. It allows synchronously get any information out of a workflow. An example of a built-in query is a stack trace of a running workflow. Cross region (in AWS terminology) replication. SWF in each region is fully independent and if the regional SWF is down all workflows in the region are stuck. Cadence supports asynchronous replication across regions. So even in the event of a complete loss of a region the workflows continue execution without interruption. Server side retry is an ability to retry an activity or a workflow according to an exponential retry policy without growing the history size. Reset is an ability to restart a workflow from any point of its execution by creating a new run and copying a part of the history. For example the reset is used to automatically roll back workflows to the point before a bad deployment that was rolled back. Cron is an ability to schedule a periodic workflow execution by passing cron string to the start method. Local activity is a short activity that is executed in the context of a decision. It uses 6x less DB operations that a normal activity execution. Long poll on history allows to efficiently watch for new history events and is also used for efficiently waiting for a workflow completion. Cadence uses the elastic search for visibility. Soon it is going to support complex searches across multiple customer defined columns which is far superior to the tag based search SWF supports. If decider constantly fails during a decision SWF records a few events on every failure eventually growing the history beyond the limit and terminating a workflow. Cadence supports transient decision feature that doesn't grow history on such failures. It allows continuing workflows without a problem after the fix to the workflow code is deployed. Cadence provides command line interface Cadence Web is open sourced and is much nicer than the SWF console. Cadence supports local development through unit testing as well as using local docker container that contains the full implementation of the Cadence service and the UI. Cadence doesn’t yet have activity and workflow type registration. The advantage is that changes to activity or workflow scheduling options do not require version bumps that affect clients.
hcnews 7y ago
Does Cadence work for low latency scenarios? Ex. web serving?
- mfateev 7y ago
  It can be used for them. There are multiple customers that have workflows with both synchronous and asynchronous components. For example customer sign up flow might require a long running background check.
- kbuckner 7y ago
  We use Cadence to run low-latency workflows for routing customer support tickets. It handles our use case very well. https://eng.uber.com/customer-obsession-ticket-routing-workflow-and-orchestration-engine/ https://eng.uber.com/customer-obsession-ticket-routing-workf...
deleted 7y ago
[deleted]
jontro 7y ago
How does this compare to netflix conductor? I've just started experimenting with it and somehow I missed this during my evaluation
- mfateev 7y ago
  I'm from Cadence team, so I'm obviously biased :). Besides very different implementation backends the main difference is that Conductor defines workflows through a JSON based DSL and Cadence defines workflows as code. Because of that it is possible to extend Cadence to interpret Conductor DSL, but the reverse is not possible. I believe that any non trivial workflows that have some state management requirements are more easily expressed as code. Any attempt to come up with JSON or XML or YAML or whatever language for workflows is always be inferior to existing programming languages like Go, Python or Java.
  - sandGorgon 7y ago
    How does this work ? Do you store the code for the workflow in the database ?
    - mfateev 7y ago
      Cadence is a service. The workflow and activity code lives outside of it. Think about Cadence workflow and activity code the same way you think about a queue consumer which is external to the queueing service.
      - sandGorgon 7y ago
        No - I'm wondering about the abstractions that allow for specific workflow specifications to happen in code versus a DSL/JSON. There are two extremes here - I can slap celery and run a bunch of custom code as workflows. On the other hand, I can use a workflow system with a built in DSL that abstracts some of the underlying behaviour out. Cadence seems to fall in the middle - and I'm wondering how it works. Why doesn't it generate to the same mess that celery+a bunch of custom python code become ?
chrischen 7y ago
Is this like Celery + RabbitMQ but with a GUI? On a high-level it sounds like that, but could someone be so kind as to give an example use case of this? How is it different that it's not just described as a distributed task queue?
- mfateev 7y ago
  The main difference is that workflow has state and tasks can be very long running. Also workflow can react to external asynchronous events. Visibility into overall progress is also a very important feature. When using a queueing system it is hard to answer about the current state of the business process. For example implementing service deployment to a public cloud usign Celery + RabbitMQ is very non trivial and error prone. It is a pretty streightforward Cadence workflow.
cle 7y ago
What does the workflow versioning story look like? This is one of the most frustrating parts of SWF that Step Functions + Lambda has effectively solved for us.
- mfateev 7y ago
  Cadence doesn't require workflow and activity type registrations which eliminates most of the problems with SWF versioning. Cadence supports versioning of the workflow code out of the box. Any change has to be protected with a version condition. It works even with shared libraries and very long running workflows.
bhouston 7y ago
How does this compare to Argo?
- mfateev 7y ago
  Argo workflows are DAGs written in YAML. The types of workfows you can create using this syntax is very limited. Cadence gives you full power of a programming language like Java or Go to implement workflow logic. It is possible to implement support for argo DSL on top of Cadence. The reverse is not possible. Cadence is also more scalable supporting tens of thousands events per second and hundreds of millions of open workflows.
timbray 7y ago
If you want to do your workflow in your own procedural code, SWF (and presumably Cadence) are good choices. If you want to use a dependency graph, Airflow is for you (but I hear operating it is kind of tricky). If you like a state-machine/flowchart kind of approach, AWS Step Functions. AWS customers these days seem to mostly like Step Functions, although SWF isn't going away, and lots of EC2 instances are running Airflow. Obviously, some people want a managed service and others want OSS that they can control & fine-tune. Nothing wrong with either choice. Most of the engineering cycles these days are going into Step Functions, keep an eye on that space.
- timbray 7y ago
  (Oh, should disclose, I helped design & build Step Functions.)
  - mfateev 7y ago
    If only SWF was extended to run deciders on AWS Lamda. Without this the main advantage of the Step Functions is hosting. I personally would rather see an integrated system where the Step Functions are a natural extension of SWF not a completely separate system. This is the direction the Cadence is going. We are planning to add support for integrating custom DSLs easily, but maintaining the core code based libraries. BTW: If anyone is interested on running the Step Functions DSL on top of Cadence contact the Cadence team. We could work together to get it implemented.
- drewda 7y ago
  Some people like both managed services _and_ OSS. It's nice to see how, say, GCP offers a managed Airflow service. (Edit: I don't mean this as a negative comment about AWS managed workflow services. Just pointing out some advantages to GCP's approach at present. I'm a happy user of both AWS and GCP services.)
whoevercares 7y ago
Interesting! Is it possible to run this with dynamoDB backend?
mleonard 7y ago
Hi I watched the Cadence talks and read through the golang code a while back and love what you're doing with cadence. Really glad to see you're moving from Thrift/TChannel to protobuf/grpc - that was a blocker before. If anyone could help me understand the following I'd appreciate it... I understand that the event history is cached at worker nodes and the whole history of events is only delivered to workers if needed (ie if out of cache) and that normally cadence manages to deliver events for the same workflow to the same worker. My question relates to what exactly happens within a single worker process. On each new event, does the worker process loop through all events in the history - starting from the beginning - in order to get back its internal state, and then process the new event, and then shutdown ready to repeat this for the next event. Or... does cadence keep the internal state around by keeping the goroutine alive waiting on a channel midway through it's workflow logic and waiting for the next event to continue execution. Thanks
- mfateev 7y ago
  It is the latter. The workflow state object is chached including the goroutines the workflow code is blocked on. And the new events are applied to the cached object.
mleonard 7y ago
The way I think about workflow engines is as follows. Please comment and correct me. Keen to discuss. Workflow engines like Cadence essentially work by letting you write regular-looking procedural logic for your workflow. This looks and feels very much like writing an async function with async-await in javascript or C#. The state in your workflow is then implicit in your code instead of explicit. Here's what I mean by that: Usually you would explicity serialise your state between each incoming event and do an atomic-compare-and-set operation on an external database to store the new state. For a new incoming event: (1) fetch the state, (2) marshal into an object in your programming language (ie a java class or golang struct), (3) given the current state, process the event and perform any external actions like sending an email. These external events need to be ok with being done with at-least-once semantics. Update the state object ready for the next incoming event. (4) serialise the state, (5) store the state in the database (atomically update with a compare-and-set operation). (6) Repeat on each event. Do everything with at-least-once repeat-on-failure semantics. In a workflow engine like cadence, what is persisted to the database is the entire history of events instead of a single state object as described above. In Cadence the code you write looks very much like async-await style code in languages like javascript or C#. The workflow logic is in some sense an async function that pauses at await statements and picks up again where it paused when the next event comes in. Remember that cadence stores the entire history of events for a workflow. It does this so that it can rerun the workflow from the beginning, this time with the new incoming event on the end of the history. Notice that you need to be careful about your workflow being deterministic. Optimisations: (1) it knows when it is replaying already-seen-events and doesn't redo external events such as sending emails. (2) it tries to resend events to the same worker node each time. It caches events at worker nodes. (3) at the macro level everything is highly-available and repeat-on-failure-with-backoff to ensure progress and at-least-once-semantics. (4) it supports repeating workflows and child workflows (5) monitoring, tracing, other things you'd expect (6) etc Importantly: notice that there is still in some sense a single state object. The state is just implicit: it is deep down in the internal state machine of the language you wrote the workflow function in (java, golang). Instead of serialising state such as 'time-since-last-email' in a state object to a database... you have 'time-since-last-email' as a local variable in the scope of the workflow function. Similarly your programming language is tracking the call stack and current execution position of the function... normally you'd keep track of progress through the workflow in the state object and condition on this state when receiving a new incoming event. Thinking about state as explicit (state object approach) versus implicit (replay-history approach) helps me when thinking about cadence and similar workflow engines. ....................................................... Thanks for reading so far. I'd love to hear from users of cadence at uber or elsewhere: (1) why do you choose to write workflows with implicit state (by replaying history) instead of storing the state explicitly as a serialised state object in the database? My guess: developer productivity of writing and maintaining the workflows. Having a common approach and single observable system for many different workflows. (2) how do you reason about long-running workflows where the business logic needs to be updated? Would this not be much easier if the state object (say a serialised protobuf) was stored explicitly in the database? (3) wouldn't non-determinism be much easier as well if you stored the state explicitly?
amelius 7y ago
Cadence is also the name of an electronics design automation company. https://www.cadence.com/ https://www.cadence.com/
viswabharathi 7y ago
Can somebody from Uber clarify? Uber could've used Cadence instead of Piper / Airflow isn't it. Please correct me if I'm wrong?
- mfateev 7y ago
  Yes, it is technically possible to use Cadence instead of Piper/Airflow. The reason the Piper/Airflow is used is mostly historical. It was around for quite a time and Cadence is a relatively new project.
onionking 7y ago
Hello, I am very interested in knowing the open source version of SWF. I am heavy SWF user and I watched all Cadence videos especially the architecture one. I wonder what is the shard recovery mechanism for a single shard on one host ? Let's say one host is down, how the shards on that host are recovered on the next host ? I heard the presenter said consistent hashing or RingPop, so I am thinking all shards should be migrated to the next available host or hosts ?

243 points by vruiz 7y ago

fokinsean 7y ago

Sounds cool, but even after poking through the repos I still don't fully understand what it does.

maxmcd 7y ago
The linked talk provides context and use cases within the first 10 minutes or so: https://atscaleconference.com/videos/cadence-microservice-architecture-beyond-requestreply/ https://atscaleconference.com/videos/cadence-microservice-ar...

dlphn___xyz 7y ago

how does this compare to Luigi or Airflow?

daveFNbuck 7y ago
Cadence says it's meant for long-running tasks, while Luigi tasks are supposed to be short.
- mfateev 7y ago
  Cadence supports both long-runnig as well as very short tasks.
  - daveFNbuck 7y ago
    I didn't think that Cadence would fail if the task finished too quickly. The point is that Cadence is designed to solve a slightly different problem than Luigi.
mfateev 7y ago
The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence. I'm from the Cadence team.
- davis_m 7y ago
  > Obviously it is also a good fit for low scale use cases. Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.
  - mfateev 7y ago
    We have both very low scale use cases as a single distributed cron as well as very high scale use cases in production at Uber.
- zimablue 7y ago
  As someone who uses (but doesn't love, actually) Airflow. I think this might be a situation where lots of people are tempted to think that they're "scale". I disagree slightly with your use of "business level" to describe this. I've seen quant hedge fund platforms(fairly intensive data engines compared to most businesses) using airflow, because when it comes down to it hundreds of DAGs is more likely than hundreds of thousands.
alfalfasprout 7y ago
I would strongly suggest not using Airflow if your company doesn't already... and that's coming from someone working at the company that made it. 1) It has no knowledge of data dependencies. This means task success is very primitive and you end up with a bunch of "polling" tasks waiting eg; for a table partition to land. 2) The UI constantly hangs and/or crashes 3) Airflow "workers" using Celery are rarely correctly given the right numbers of tasks. OOM-ing, etc. are all commonplace even if using Docker. And many, many more. I strongly suggest using Apache Beam or Argo w/ Kubernetes instead. They can scale quite a bit more and deal with long running tasks well.
- red_hare 7y ago
  I'm actually finding airflow + beam + kubernetes to be a really powerful combination. Especially if you're on GCP where airflow is managed by Cloud Composer, kubernetes is managed by GKE, and beam is managed by Dataflow. Our airflow cluster _does_ almost nothing. Our operators just tell kubernetes to run containers with commands or dataflow to run some beam template and waits for the results from afar. I love this setup but man do I agree with your points about the UI. For a product so powerful I can't believe how problematic it is.

formalsystem 7y ago

Does anyone know what are the best OSS Orchestration Engines? Am wondering what I should be comparing this to.

manyxcxi 7y ago
I don’t know about best, as there are lots of trade offs between various projects. I’ve reviewed and worked with a lot of Java based ones, top of mind: - Airflow - jBPM - Enhydra Shark - Activiti - Netflix Conductor Most of those are more workflow engine than pure orchestration. A few years back, we searched high and low and I was generally unhappy with the commercial offerings so we wound up building around Netflix Conductor (after a disastrous run with Joget, which is built on Enhydra Shark). Since then I’ve been pretty happy with Conductor, submitted numerous PRs and accidentally became one of the people keeping the MySQL backend implementation going forward.
- mfateev 7y ago
  AFAIK Netflix Conductor was inspired by the AWS Simple Workflow engine. The Cadence is a direct evolution of SWF in the open source world. I was tech lead for both of them :).
- opportune 7y ago
  I guess technically Oozie is also a FOSS Java-based workflow/orchestration engine, except it's mostly for Hadoop jobs. Haven't used too many orchestration engines but I found Oozie very barebones
- zok3102 7y ago
  You may also want to look at Project Flogo (flogo.io) - in particular, the Flogo Flow action. It's a process engine in Golang that can be used to implement service orchestration, event-driven choreography integration patterns. Flogo has been focused more on stateless service choreography running in Serverless FaaS or Edge devices - however the underlying process engine can be used to implement a long running orchestration process with fully externalized state for resilience and scale. It's 3-clause BSD including the modeling UI with hundreds of triggers and activity extensions in the community. We have a fairly active gitter channel in case you have questions for the core team (https://gitter.im/project-flogo/Lobby https://gitter.im/project-flogo/Lobby) Disclaimer: I work at TIBCO & we provide commercial support for Project Flogo as well as use it in our commercial products.

redact207 7y ago

I've been working on a similar workflow engine for node at https://www.npmjs.com/package/@node-ts/bus-workflow https://www.npmjs.com/package/@node-ts/bus-workflow The main objective of workflows is to manage long running processes. By processes I mean business processes like coordinating the activities of fulfilling a customer order (settling charges, picking inventory, packing, dispatching, email receipts etc). It's a way to keep all those individual commands decoupled but coordinate them at a higher level. This isn't a new concept by any means, and is often paired with Domain Driven Design and message based systems. Doing so gives you a library of events everytime something happens in your system that can be reacted to in a workflow. If you've ever dealt with microservices, or even a monolith where two internal services are incorrectly coupled together then this approach may be worth looking into.

wetpaste 7y ago
Thank you. I feel kind of silly about this but I feel like I've had a hard time understanding when an org should, or could use something like this. I have seen them mentioned but every time it's explained it's explained with more abstract language on top of it that confuses me. I keep hearing "it manages business processes" but then it fails to mention if this means like, a human being's process within an org, or something coupled with an application of some sort that has business processes in the application? Does this type of thing replace sort of what Jira does, make a ticket and then pass it off to the next team or whatever? Do you ship it with the app for on-premise deployments of a software product? I have a hard time seeing the big picture with things like this sometimes. Then I hear workflow orchestrator and I think, oh okay so like ansible, but for, work...flows? But what is a workflow really exactly?
- mfateev 7y ago
  My personal opinion is that the workflow is any business logic that goes beyond a single request reply. Examples of workflows: * Service deployment. * Uber trip * Media processing (download file, transcode, upload result) * Order processing * Customer support ticket processing * Customer incentive program management * Data pipeline processing * ML Training * Distributed CRON * Customer signup flow and many others.

ajbosco 7y ago

I thought they used Piper (based on Airflow) for workflows at Uber. https://eng.uber.com/managing-data-workflows-at-scale/ https://eng.uber.com/managing-data-workflows-at-scale/

thor24 7y ago
That is for data workflows (your etl jobs basically). This is for services.
- dgladkov 7y ago
  If you already have a service that requires ETL, Cadence might be a good choice as well. You will need more boilerplate and setup compared to Airflow/Piper, but you can share data structures between ETLs and services that use it and have more control over execution and deployment as you own your workers.

vxa_victor 7y ago

How does it compare to BPM solution like Camunda?

mfateev 7y ago
Cadence allows to write workflows as code. Think of it as a virtual machine for OO code that makes that code fully fault tolerant to process failures. As it is code it can be used to implement any business logic. Camunda is BPMN engine which interprets BPMN workflow definition. It is possible to implement a Cadence workflow that interprets BPMN without changing the core Cadence service.
- _ph_ 7y ago
  And we are talking about a software company and a software package. Sounds like a strong overlap to me - or do you think you can call a new software library "microsoft"?

bozoUser 7y ago

Looking at a few comments, can someone answer on whats are the nuances between managing data workflows vs service workflows ?

mshockwave 7y ago

just a random comment: I thought Cadence, the EDA company, owns the trademark for the name

jacques_chester 7y ago
Trademarks are partitioned by subject matter: https://www.uspto.gov/web/patents/classification/selectnumwithtitle.htm https://www.uspto.gov/web/patents/classification/selectnumwi...

iblaine 7y ago

Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours. Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.

mfateev 7y ago
Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.
- iblaine 7y ago
  Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?
  - mfateev 7y ago
    For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.
    - chrischen 7y ago
      For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events? Also how does it listen for events? From polling?
      - mfateev 7y ago

deleted 7y ago

[deleted]

jbbarth 7y ago

Cadence looks like an OSS version of Amazon Simple Workflow (SWF) service. The author used to work on SWF at AWS afaik. I'm a heavy SWF user at work for managing complex data pipelines. SWF requires an important conceptual and tooling effort in the beginning, but it gets reimbursed if you use it a lot. As for other comments mentioning Airflow: the programming model is quite different , since Airflow as far as I understand forces to provide a DAG of tasks upfront. SWF (and Cadence?) doesn't, it coordinates the work of Deciders and Activity Workers and only acts as a source of truth for the state of the workflow (+ distribute task in a unique manner to many long-polling workers). As a result you don't declare anything upfront and can have deciders take dynamic decisions along the way, which is really nice when you want very dynamic logic for your workflows (e.g. dynamic partitioning of tasks, decisions depending on external factors, etc.). I'd love to have Maxim insights about how Cadence compares to SWF, and what would be the reasons/challenges behind migrating from SWF to Cadence for SWF users (except that SWF is basically stale for 4+ years and rigged with arbitrary limits)

mfateev 7y ago
Cadence vs SWF Cadence was conceived and is still led by the original tech leads of the SWF. SWF had no new features added for the last 5 years. Cadence is open sourced and is under active development. Cadence was initially based on SWF public API. It uses Thrift and TChannel for communication and SWF uses AWS version of REST. Currently the API is not compatible with SWF as Cadence added a large number of new features and deprecated a few problematic ones. We are planning migrating to gRPC later this year. Cadence can potentially run on any database that supports single shard multi-row transactions as a backend. Currently it supports Cassandra and MySQL. SWF has pretty tight throttling limits. Cadence scales very well with use cases in production that require 100s of millions of open workflows and thousands of events per second. SWF has pretty tight limits on individual payloads and number of events. For example maximum activity input size is 32k. Cadence currently has 256k limit. SWF history size limit is 10k events while Cadence limit 200k. All other limits are also higher. Cadence has no limit on the activity and workflow execution duration. Cadence through archival supports unlimited retention after a workflow closure. SWF has Java and Ruby client libraries. Cadence has Java and Go client libraries. SWF Java library is fully asynchronous and relies on both code generation (through annotation processor) and AspectJ. It is hard to set up, doesn't play well with IDEs and has very steep learning curve. Cadence Java library (as well as Go one) allow writing workflows as synchronous programs which greatly simplifies the programming model. It also just a library without any need for code generation or AspectJ or similar intrusive technologies. Cadence client side libraries have much better unit testing support. For example the Java library utilizes an in-memory implementation of the Cadence service. Cadence features that SWF doesn't have: Workflow stickiness. SWF replays the whole workflow history on every decision. Which means that a workflow resource usage is proportional to O(n*n) of number of events in the history. Cadence caches workflows on a worker and delivers only new events to them. The whole history is replayed only when a worker goes down or the workflow gets out of cache. So Cadence workflow resource usage is O(n) of number of events in the history. For large workflows it makes a huge difference. It also leads to higher per workflow scale. For example it is not recommended to have workflows that execute over a hundred activities in SWF. Cadence routinely executes workflows that have over thousand activities or child workflows. Query workflow execution. It allows synchronously get any information out of a workflow. An example of a built-in query is a stack trace of a running workflow. Cross region (in AWS terminology) replication. SWF in each region is fully independent and if the regional SWF is down all workflows in the region are stuck. Cadence supports asynchronous replication across regions. So even in the event of a complete loss of a region the workflows continue execution without interruption. Server side retry is an ability to retry an activity or a workflow according to an exponential retry policy without growing the history size. Reset is an ability to restart a workflow from any point of its execution by creating a new run and copying a part of the history. For example the reset is used to automatically roll back workflows to the point before a bad deployment that was rolled back. Cron is an ability to schedule a periodic workflow execution by passing cron string to the start method. Local activity is a short activity that is executed in the context of a decision. It uses 6x less DB operations that a normal activity execution. Long poll on history allows to efficiently watch for new history events and is also used for efficiently waiting for a workflow completion. Cadence uses the elastic search for visibility. Soon it is going to support complex searches across multiple customer defined columns which is far superior to the tag based search SWF supports. If decider constantly fails during a decision SWF records a few events on every failure eventually growing the history beyond the limit and terminating a workflow. Cadence supports transient decision feature that doesn't grow history on such failures. It allows continuing workflows without a problem after the fix to the workflow code is deployed. Cadence provides command line interface Cadence Web is open sourced and is much nicer than the SWF console. Cadence supports local development through unit testing as well as using local docker container that contains the full implementation of the Cadence service and the UI. Cadence doesn’t yet have activity and workflow type registration. The advantage is that changes to activity or workflow scheduling options do not require version bumps that affect clients.

hcnews 7y ago

Does Cadence work for low latency scenarios? Ex. web serving?

mfateev 7y ago
It can be used for them. There are multiple customers that have workflows with both synchronous and asynchronous components. For example customer sign up flow might require a long running background check.
kbuckner 7y ago
We use Cadence to run low-latency workflows for routing customer support tickets. It handles our use case very well. https://eng.uber.com/customer-obsession-ticket-routing-workflow-and-orchestration-engine/ https://eng.uber.com/customer-obsession-ticket-routing-workf...

deleted 7y ago

[deleted]

jontro 7y ago

How does this compare to netflix conductor? I've just started experimenting with it and somehow I missed this during my evaluation

mfateev 7y ago
I'm from Cadence team, so I'm obviously biased :). Besides very different implementation backends the main difference is that Conductor defines workflows through a JSON based DSL and Cadence defines workflows as code. Because of that it is possible to extend Cadence to interpret Conductor DSL, but the reverse is not possible. I believe that any non trivial workflows that have some state management requirements are more easily expressed as code. Any attempt to come up with JSON or XML or YAML or whatever language for workflows is always be inferior to existing programming languages like Go, Python or Java.
- sandGorgon 7y ago
  How does this work ? Do you store the code for the workflow in the database ?
  - mfateev 7y ago
    Cadence is a service. The workflow and activity code lives outside of it. Think about Cadence workflow and activity code the same way you think about a queue consumer which is external to the queueing service.
    - sandGorgon 7y ago
      No - I'm wondering about the abstractions that allow for specific workflow specifications to happen in code versus a DSL/JSON. There are two extremes here - I can slap celery and run a bunch of custom code as workflows. On the other hand, I can use a workflow system with a built in DSL that abstracts some of the underlying behaviour out. Cadence seems to fall in the middle - and I'm wondering how it works. Why doesn't it generate to the same mess that celery+a bunch of custom python code become ?

chrischen 7y ago

Is this like Celery + RabbitMQ but with a GUI? On a high-level it sounds like that, but could someone be so kind as to give an example use case of this? How is it different that it's not just described as a distributed task queue?

mfateev 7y ago
The main difference is that workflow has state and tasks can be very long running. Also workflow can react to external asynchronous events. Visibility into overall progress is also a very important feature. When using a queueing system it is hard to answer about the current state of the business process. For example implementing service deployment to a public cloud usign Celery + RabbitMQ is very non trivial and error prone. It is a pretty streightforward Cadence workflow.

cle 7y ago

What does the workflow versioning story look like? This is one of the most frustrating parts of SWF that Step Functions + Lambda has effectively solved for us.

mfateev 7y ago
Cadence doesn't require workflow and activity type registrations which eliminates most of the problems with SWF versioning. Cadence supports versioning of the workflow code out of the box. Any change has to be protected with a version condition. It works even with shared libraries and very long running workflows.

bhouston 7y ago

How does this compare to Argo?

mfateev 7y ago
Argo workflows are DAGs written in YAML. The types of workfows you can create using this syntax is very limited. Cadence gives you full power of a programming language like Java or Go to implement workflow logic. It is possible to implement support for argo DSL on top of Cadence. The reverse is not possible. Cadence is also more scalable supporting tens of thousands events per second and hundreds of millions of open workflows.

timbray 7y ago

If you want to do your workflow in your own procedural code, SWF (and presumably Cadence) are good choices. If you want to use a dependency graph, Airflow is for you (but I hear operating it is kind of tricky). If you like a state-machine/flowchart kind of approach, AWS Step Functions. AWS customers these days seem to mostly like Step Functions, although SWF isn't going away, and lots of EC2 instances are running Airflow. Obviously, some people want a managed service and others want OSS that they can control & fine-tune. Nothing wrong with either choice. Most of the engineering cycles these days are going into Step Functions, keep an eye on that space.

timbray 7y ago
(Oh, should disclose, I helped design & build Step Functions.)
- mfateev 7y ago
  If only SWF was extended to run deciders on AWS Lamda. Without this the main advantage of the Step Functions is hosting. I personally would rather see an integrated system where the Step Functions are a natural extension of SWF not a completely separate system. This is the direction the Cadence is going. We are planning to add support for integrating custom DSLs easily, but maintaining the core code based libraries. BTW: If anyone is interested on running the Step Functions DSL on top of Cadence contact the Cadence team. We could work together to get it implemented.
drewda 7y ago
Some people like both managed services _and_ OSS. It's nice to see how, say, GCP offers a managed Airflow service. (Edit: I don't mean this as a negative comment about AWS managed workflow services. Just pointing out some advantages to GCP's approach at present. I'm a happy user of both AWS and GCP services.)

whoevercares 7y ago

Interesting! Is it possible to run this with dynamoDB backend?

mleonard 7y ago

Hi I watched the Cadence talks and read through the golang code a while back and love what you're doing with cadence. Really glad to see you're moving from Thrift/TChannel to protobuf/grpc - that was a blocker before. If anyone could help me understand the following I'd appreciate it... I understand that the event history is cached at worker nodes and the whole history of events is only delivered to workers if needed (ie if out of cache) and that normally cadence manages to deliver events for the same workflow to the same worker. My question relates to what exactly happens within a single worker process. On each new event, does the worker process loop through all events in the history - starting from the beginning - in order to get back its internal state, and then process the new event, and then shutdown ready to repeat this for the next event. Or... does cadence keep the internal state around by keeping the goroutine alive waiting on a channel midway through it's workflow logic and waiting for the next event to continue execution. Thanks

mfateev 7y ago
It is the latter. The workflow state object is chached including the goroutines the workflow code is blocked on. And the new events are applied to the cached object.

mleonard 7y ago

The way I think about workflow engines is as follows. Please comment and correct me. Keen to discuss. Workflow engines like Cadence essentially work by letting you write regular-looking procedural logic for your workflow. This looks and feels very much like writing an async function with async-await in javascript or C#. The state in your workflow is then implicit in your code instead of explicit. Here's what I mean by that: Usually you would explicity serialise your state between each incoming event and do an atomic-compare-and-set operation on an external database to store the new state. For a new incoming event: (1) fetch the state, (2) marshal into an object in your programming language (ie a java class or golang struct), (3) given the current state, process the event and perform any external actions like sending an email. These external events need to be ok with being done with at-least-once semantics. Update the state object ready for the next incoming event. (4) serialise the state, (5) store the state in the database (atomically update with a compare-and-set operation). (6) Repeat on each event. Do everything with at-least-once repeat-on-failure semantics. In a workflow engine like cadence, what is persisted to the database is the entire history of events instead of a single state object as described above. In Cadence the code you write looks very much like async-await style code in languages like javascript or C#. The workflow logic is in some sense an async function that pauses at await statements and picks up again where it paused when the next event comes in. Remember that cadence stores the entire history of events for a workflow. It does this so that it can rerun the workflow from the beginning, this time with the new incoming event on the end of the history. Notice that you need to be careful about your workflow being deterministic. Optimisations: (1) it knows when it is replaying already-seen-events and doesn't redo external events such as sending emails. (2) it tries to resend events to the same worker node each time. It caches events at worker nodes. (3) at the macro level everything is highly-available and repeat-on-failure-with-backoff to ensure progress and at-least-once-semantics. (4) it supports repeating workflows and child workflows (5) monitoring, tracing, other things you'd expect (6) etc Importantly: notice that there is still in some sense a single state object. The state is just implicit: it is deep down in the internal state machine of the language you wrote the workflow function in (java, golang). Instead of serialising state such as 'time-since-last-email' in a state object to a database... you have 'time-since-last-email' as a local variable in the scope of the workflow function. Similarly your programming language is tracking the call stack and current execution position of the function... normally you'd keep track of progress through the workflow in the state object and condition on this state when receiving a new incoming event. Thinking about state as explicit (state object approach) versus implicit (replay-history approach) helps me when thinking about cadence and similar workflow engines. ....................................................... Thanks for reading so far. I'd love to hear from users of cadence at uber or elsewhere: (1) why do you choose to write workflows with implicit state (by replaying history) instead of storing the state explicitly as a serialised state object in the database? My guess: developer productivity of writing and maintaining the workflows. Having a common approach and single observable system for many different workflows. (2) how do you reason about long-running workflows where the business logic needs to be updated? Would this not be much easier if the state object (say a serialised protobuf) was stored explicitly in the database? (3) wouldn't non-determinism be much easier as well if you stored the state explicitly?

amelius 7y ago

Cadence is also the name of an electronics design automation company. https://www.cadence.com/ https://www.cadence.com/

viswabharathi 7y ago

Can somebody from Uber clarify? Uber could've used Cadence instead of Piper / Airflow isn't it. Please correct me if I'm wrong?

mfateev 7y ago
Yes, it is technically possible to use Cadence instead of Piper/Airflow. The reason the Piper/Airflow is used is mostly historical. It was around for quite a time and Cadence is a relatively new project.

onionking 7y ago

Hello, I am very interested in knowing the open source version of SWF. I am heavy SWF user and I watched all Cadence videos especially the architecture one. I wonder what is the shard recovery mechanism for a single shard on one host ? Let's say one host is down, how the shards on that host are recovered on the next host ? I heard the presenter said consistent hashing or RingPop, so I am thinking all shards should be migrated to the next available host or hosts ?