6 ms·

The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good

The author seems to think that Dagster or Prefect will take over Airflow, I don't think this is true. All of them being open source means that if one has a good idea or better way of doing something the other can quickly implement the feature and even use the some of same code. We saw it with Airflow implementing the TaskFlow API as a response to Dagster, and in a few weeks Airflow 2.4 is going to have dataset scheduling released. So, Airflow's head start is going to be extremely hard to over come, if they remain adaptable. Also, as other's have mentioned the comparison to MWAA is unfair, the true own of Airflow is Astronomer as they have over 50% of the commits to Airflow, and Astronomer is a much better product the MWAA.

frasermarlow 4y ago
For sure, between these competing products you will see one respond to a popular feature by mimicking it. But the Dagster framework is fundamentally different from Airflow so over time I don't see it being a feature-vs-feature decision but more of a declarative/reconciliation vs. imperative/task-centric split.
fulafel 4y ago
Is open source really a big factor? There's no magic sauce in the implementation in this kind of stuff, and you can't just copy and paste code over.
sryza 4y ago
Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs. Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.
- KptMarchewa 4y ago
  Spark, after all _is_ a large Apache project?
  - zyd14 4y ago
    probably meant databricks
  - sryza 4y ago
    Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors. Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
  - glogla 4y ago
    It is a bit more nuanced. On one hand, it is immensely successful Apache project. On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
    - KptMarchewa 4y ago
      >On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. Well, can't disagree with that. >There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition. Any link to that? I can't find the story.
      - 4y ago
DSJayatillake 4y ago
I don't really think that either will overtake Airflow in terms of sheer number or users or orgs it is deployed at. 1. Dagster and Prefect won't ever have a hosted version on AWS/GCP (teams can spin these up from their consoles without even considering a vendor conversation or leaving their infra). 2. There are very large Airflow projects out there with huge DE teams running them, that just can't really move to Dagster or Prefect very easily. Not do they really want to if they can do what they need to in a tool they are familiar with. My perspectives have been very much coming at the tools with fresh eyes. Will they overtake Airflow in terms of features? They probably already have.
- ciguy 4y ago
  Dagster already offers Dagster Cloud which came out a month or so ago[1]. It's a fully hosted solution for teams that don't want to run their own. [1] https://dagster.io/cloud https://dagster.io/cloud

frasermarlow 4y ago

For sure, between these competing products you will see one respond to a popular feature by mimicking it. But the Dagster framework is fundamentally different from Airflow so over time I don't see it being a feature-vs-feature decision but more of a declarative/reconciliation vs. imperative/task-centric split.

fulafel 4y ago

Is open source really a big factor? There's no magic sauce in the implementation in this kind of stuff, and you can't just copy and paste code over.

sryza 4y ago

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs. Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.

KptMarchewa 4y ago
Spark, after all _is_ a large Apache project?
- zyd14 4y ago
  probably meant databricks
- sryza 4y ago
  Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors. Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
- glogla 4y ago
  It is a bit more nuanced. On one hand, it is immensely successful Apache project. On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
  - KptMarchewa 4y ago
    >On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. Well, can't disagree with that. >There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition. Any link to that? I can't find the story.
    - 4y ago

DSJayatillake 4y ago

I don't really think that either will overtake Airflow in terms of sheer number or users or orgs it is deployed at. 1. Dagster and Prefect won't ever have a hosted version on AWS/GCP (teams can spin these up from their consoles without even considering a vendor conversation or leaving their infra). 2. There are very large Airflow projects out there with huge DE teams running them, that just can't really move to Dagster or Prefect very easily. Not do they really want to if they can do what they need to in a tool they are familiar with. My perspectives have been very much coming at the tools with fresh eyes. Will they overtake Airflow in terms of features? They probably already have.

ciguy 4y ago
Dagster already offers Dagster Cloud which came out a month or so ago[1]. It's a fully hosted solution for teams that don't want to run their own. [1] https://dagster.io/cloud https://dagster.io/cloud