3 ms·

Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically process

Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours. Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.

mfateev 7y ago
Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.
- iblaine 7y ago
  Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?
  - mfateev 7y ago
    For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.
    - chrischen 7y ago
      For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events? Also how does it listen for events? From polling?
      - mfateev 7y ago
        It could be solved by Celery, but it would also require a database and Celery doesn't scale that well unless runs on top of Redis which is not really fault tolerant. Also actions invocation with guarantees and exponential retries is not trivial with Celery. Actually at Uber large number of services are being migrated from Python/Celery to Go/Cadence.
- viswabharathi 7y ago
  Uber could've used Cadence in place of Piper/Airflow isn't it. Can you shed more light on where both of these fit.

mfateev 7y ago

Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.

iblaine 7y ago
Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?
- mfateev 7y ago
  For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.
  - chrischen 7y ago
    For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events? Also how does it listen for events? From polling?
    - mfateev 7y ago
      It could be solved by Celery, but it would also require a database and Celery doesn't scale that well unless runs on top of Redis which is not really fault tolerant. Also actions invocation with guarantees and exponential retries is not trivial with Celery. Actually at Uber large number of services are being migrated from Python/Celery to Go/Cadence.
viswabharathi 7y ago
Uber could've used Cadence in place of Piper/Airflow isn't it. Can you shed more light on where both of these fit.