4 ms·

how does this compare to Luigi or Airflow?

by dlphn___xyz 8y ago

daveFNbuck 8y ago
Cadence says it's meant for long-running tasks, while Luigi tasks are supposed to be short.
- mfateev 8y ago
  Cadence supports both long-runnig as well as very short tasks.
  - daveFNbuck 7y ago
    I didn't think that Cadence would fail if the task finished too quickly. The point is that Cadence is designed to solve a slightly different problem than Luigi.
mfateev 8y ago
The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence. I'm from the Cadence team.
- davis_m 8y ago
  > Obviously it is also a good fit for low scale use cases. Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.
  - mfateev 8y ago
    We have both very low scale use cases as a single distributed cron as well as very high scale use cases in production at Uber.
- zimablue 7y ago
  As someone who uses (but doesn't love, actually) Airflow. I think this might be a situation where lots of people are tempted to think that they're "scale". I disagree slightly with your use of "business level" to describe this. I've seen quant hedge fund platforms(fairly intensive data engines compared to most businesses) using airflow, because when it comes down to it hundreds of DAGs is more likely than hundreds of thousands.
alfalfasprout 8y ago
I would strongly suggest not using Airflow if your company doesn't already... and that's coming from someone working at the company that made it. 1) It has no knowledge of data dependencies. This means task success is very primitive and you end up with a bunch of "polling" tasks waiting eg; for a table partition to land. 2) The UI constantly hangs and/or crashes 3) Airflow "workers" using Celery are rarely correctly given the right numbers of tasks. OOM-ing, etc. are all commonplace even if using Docker. And many, many more. I strongly suggest using Apache Beam or Argo w/ Kubernetes instead. They can scale quite a bit more and deal with long running tasks well.
- red_hare 8y ago
  I'm actually finding airflow + beam + kubernetes to be a really powerful combination. Especially if you're on GCP where airflow is managed by Cloud Composer, kubernetes is managed by GKE, and beam is managed by Dataflow. Our airflow cluster _does_ almost nothing. Our operators just tell kubernetes to run containers with commands or dataflow to run some beam template and waits for the results from afar. I love this setup but man do I agree with your points about the UI. For a product so powerful I can't believe how problematic it is.

how does this compare to Luigi or Airflow?

daveFNbuck 8y ago

Cadence says it's meant for long-running tasks, while Luigi tasks are supposed to be short.

mfateev 8y ago
Cadence supports both long-runnig as well as very short tasks.
- daveFNbuck 7y ago
  I didn't think that Cadence would fail if the task finished too quickly. The point is that Cadence is designed to solve a slightly different problem than Luigi.

mfateev 8y ago

The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence. I'm from the Cadence team.

davis_m 8y ago
> Obviously it is also a good fit for low scale use cases. Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.
- mfateev 8y ago
  We have both very low scale use cases as a single distributed cron as well as very high scale use cases in production at Uber.
zimablue 7y ago
As someone who uses (but doesn't love, actually) Airflow. I think this might be a situation where lots of people are tempted to think that they're "scale". I disagree slightly with your use of "business level" to describe this. I've seen quant hedge fund platforms(fairly intensive data engines compared to most businesses) using airflow, because when it comes down to it hundreds of DAGs is more likely than hundreds of thousands.

alfalfasprout 8y ago

I would strongly suggest not using Airflow if your company doesn't already... and that's coming from someone working at the company that made it. 1) It has no knowledge of data dependencies. This means task success is very primitive and you end up with a bunch of "polling" tasks waiting eg; for a table partition to land. 2) The UI constantly hangs and/or crashes 3) Airflow "workers" using Celery are rarely correctly given the right numbers of tasks. OOM-ing, etc. are all commonplace even if using Docker. And many, many more. I strongly suggest using Apache Beam or Argo w/ Kubernetes instead. They can scale quite a bit more and deal with long running tasks well.

red_hare 8y ago
I'm actually finding airflow + beam + kubernetes to be a really powerful combination. Especially if you're on GCP where airflow is managed by Cloud Composer, kubernetes is managed by GKE, and beam is managed by Dataflow. Our airflow cluster _does_ almost nothing. Our operators just tell kubernetes to run containers with commands or dataflow to run some beam template and waits for the results from afar. I love this setup but man do I agree with your points about the UI. For a product so powerful I can't believe how problematic it is.