4 ms·

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially look

by sryza 4y ago

Dagster contributor here responding to Astronomer employee ^^. A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different: - TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small. - If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs. Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.

KptMarchewa 4y ago
Spark, after all _is_ a large Apache project?
- zyd14 4y ago
  probably meant databricks
- sryza 4y ago
  Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors. Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
- glogla 4y ago
  It is a bit more nuanced. On one hand, it is immensely successful Apache project. On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
  - KptMarchewa 4y ago
    >On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. Well, can't disagree with that. >There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition. Any link to that? I can't find the story.
    - glogla 4y ago
      > Any link to that? I can't find the story. I can't find it either, I'm pretty sure I heard it on some conference, so consider it made up. It was years ago anyway.

by sryza 4y ago

KptMarchewa 4y ago

Spark, after all _is_ a large Apache project?

zyd14 4y ago
probably meant databricks
sryza 4y ago
Yeah, I don't mean to knock large Apache projects in general - they can be great software. My point is that it's hard for them to change / successfully copy their competitors. Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
glogla 4y ago
It is a bit more nuanced. On one hand, it is immensely successful Apache project. On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
- KptMarchewa 4y ago
  >On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players. Well, can't disagree with that. >There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition. Any link to that? I can't find the story.
  - glogla 4y ago
    > Any link to that? I can't find the story. I can't find it either, I'm pretty sure I heard it on some conference, so consider it made up. It was years ago anyway.