5 ms·

There're so many alternatives to Airflow nowadays that you really need to make sure that Airflow is the best solution (or even a solution) to your use case. The

There're so many alternatives to Airflow nowadays that you really need to make sure that Airflow is the best solution (or even a solution) to your use case. There's plenty of use cases better resolved with tools like Prefect or Dagster, but I suppose the inertia to install the tool everyone knows about is really big. BTW, here's https://github.com/pditommaso/awesome-pipeline https://github.com/pditommaso/awesome-pipeline a list of almost 200 pipeline toolkits.

Peteris 6y ago
One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.
- sdepablos 6y ago
  Interesting (and in the list). Reading the documentation looks like the only other format it allows to be exported is Airflow, isn't it?
dmayle 6y ago
I've gone through a large number of these, and I think that Airflow is the best on Kubernetes for managed orchestration. The things I like are: * Source control for workflows/DAGs (using git-sync) * Tracking/retries with SLAs * Jobs run in Kubernetes * Web UI for management * Fully open source I also use Argo Workflows, because I like its native handling of Kubernetes objects (e.g. the ability to manage and update a deployment as one of the steps), but it just doesn't have the orchestration/tracking side of things very well managed yet
- tiagod 6y ago
  I absolutely love Cadence/Temporal. It can do anything the others can do (and you could implement any other engine with it), without any DSLs.
  - silasb 6y ago
    Just found out about Temporal and it looks interesting. I eager to jump in but our organization primarily uses Ruby. I know the big difference between Cadence vs Temporal is the fact they are using GRPC which seems much easier to adopt.
    - ttsda 6y ago
      I also find the API to be more consistent on Temporal (I began implementing a project in Cadence before finding out about Temporal)
    - mfateev 6y ago
      Temporal has an externally contributed Ruby client. It is going to be open sourced very soon.
      - silasb 6y ago
        Very exciting news. Is there a way that I can subscribe to get info on when it'll be released?
        mfateev 6y ago
        There is #ruby-sdk channel in the Temporal Slack: https://join.slack.com/t/temporalio/shared_invite/zt-c1e99p8g-beF7~ZZW2HP6gGStXD8Nuw https://join.slack.com/t/temporalio/shared_invite/zt-c1e99p8...
  - deleted 6y ago
    [deleted]
  - cpdean 6y ago
    I think this is the link to cadence: https://github.com/uber/cadence https://github.com/uber/cadence and temporal: https://github.com/temporalio/temporal https://github.com/temporalio/temporal
    - 6y ago
walleeee 6y ago
I've had a wonderful experience with Dagster so far. I love that it can deploy to Airflow, Celery, Dask, etc, I love the Dagit server and UI and that I can orchestrate pipelines over HTTP, I love the notebook integration via Papermill, I love that it's all free (looking at Prefect here...), and the team is extremely responsive on both Slack and GitHub
- _frkl 6y ago
  Didn't Prefect open source their orchestration component recently, or am I mistaken? What part of Prefect is still closed?
  - walleeee 6y ago
    Oh, I was saying it wasn't free. I think you're right and it is fully open source
    - rywalker 6y ago
      It's not quite "open source", better labelled as "source available"
eyeball 6y ago
We used Luigi because airflow was to complicated to get an unsupportive IT department to install.
adolph 6y ago
Here's a nice podcast about Prefect that talks about its and Airflow's approachs: https://softwareengineeringdaily.com/2020/04/29/prefect-dataflow-scheduler-with-jeremiah-lowin/ https://softwareengineeringdaily.com/2020/04/29/prefect-data...
sequoia 6y ago
> A curated list of awesome pipeline toolkits Is this "curated"? It seems like an exhaustive "dump" of toolkits.
- unixhero 6y ago
  What curation do you feel it lacks?
  - sequoia 6y ago
    To curate a collection is to be an editor, determining what to include and what to exclude. From a curated selection of tools, I expect to see a selection of the best tools, chosen by a knowledgable curator who has evaluated the tools in some way. So for example, if you said "I want to start a library, donate any books you have at my house" that would not be a "curated" collection in my opionion. If you went through & evaluated the books, selecting only those that you'd personally recommend and discarding the rest, that would be a "curated" collection. (Strictly speaking any maintenance of a collection even just "dump everything at my door at I'll put it on the pile" could be considered "curation" but to call a list of tools "curated" suggests there's some selection going on, and there does not appear to be on this list.) EDIT: In fact, all of the definitions of "curate" here[0] start with the word "select," for example "Select, organize, and look after the items in (a collection or exhibition)." This fits my meaning. This definition[1] of "curator" requires one to merely have "care and superintendence of something," so in that sense the list is "curated" insofar as someone looks at PRs and clicks the "merge" button. The Miriam-Webster definition, however, speaks of curating a more general "something" rather than a "collection." If you "curate" a statue by cleaning & protecting it, fine, you don't need to select the statue. However, when a collection is curated, in my opinion, this necessarily implies selection, not just maintenance. 0: https://www.lexico.com/en/definition/curate https://www.lexico.com/en/definition/curate 1: https://www.merriam-webster.com/dictionary/curator https://www.merriam-webster.com/dictionary/curator

Peteris 6y ago

One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.

sdepablos 6y ago
Interesting (and in the list). Reading the documentation looks like the only other format it allows to be exported is Airflow, isn't it?

dmayle 6y ago

I've gone through a large number of these, and I think that Airflow is the best on Kubernetes for managed orchestration. The things I like are: * Source control for workflows/DAGs (using git-sync) * Tracking/retries with SLAs * Jobs run in Kubernetes * Web UI for management * Fully open source I also use Argo Workflows, because I like its native handling of Kubernetes objects (e.g. the ability to manage and update a deployment as one of the steps), but it just doesn't have the orchestration/tracking side of things very well managed yet

tiagod 6y ago
I absolutely love Cadence/Temporal. It can do anything the others can do (and you could implement any other engine with it), without any DSLs.
- silasb 6y ago
  Just found out about Temporal and it looks interesting. I eager to jump in but our organization primarily uses Ruby. I know the big difference between Cadence vs Temporal is the fact they are using GRPC which seems much easier to adopt.
  - ttsda 6y ago
    I also find the API to be more consistent on Temporal (I began implementing a project in Cadence before finding out about Temporal)
  - mfateev 6y ago
    Temporal has an externally contributed Ruby client. It is going to be open sourced very soon.
    - silasb 6y ago
      Very exciting news. Is there a way that I can subscribe to get info on when it'll be released?
      - mfateev 6y ago
        There is #ruby-sdk channel in the Temporal Slack: https://join.slack.com/t/temporalio/shared_invite/zt-c1e99p8g-beF7~ZZW2HP6gGStXD8Nuw https://join.slack.com/t/temporalio/shared_invite/zt-c1e99p8...
- deleted 6y ago
  [deleted]
- cpdean 6y ago
  I think this is the link to cadence: https://github.com/uber/cadence https://github.com/uber/cadence and temporal: https://github.com/temporalio/temporal https://github.com/temporalio/temporal
  - 6y ago

walleeee 6y ago

I've had a wonderful experience with Dagster so far. I love that it can deploy to Airflow, Celery, Dask, etc, I love the Dagit server and UI and that I can orchestrate pipelines over HTTP, I love the notebook integration via Papermill, I love that it's all free (looking at Prefect here...), and the team is extremely responsive on both Slack and GitHub

_frkl 6y ago
Didn't Prefect open source their orchestration component recently, or am I mistaken? What part of Prefect is still closed?
- walleeee 6y ago
  Oh, I was saying it wasn't free. I think you're right and it is fully open source
  - rywalker 6y ago
    It's not quite "open source", better labelled as "source available"

eyeball 6y ago

We used Luigi because airflow was to complicated to get an unsupportive IT department to install.

adolph 6y ago

Here's a nice podcast about Prefect that talks about its and Airflow's approachs: https://softwareengineeringdaily.com/2020/04/29/prefect-dataflow-scheduler-with-jeremiah-lowin/ https://softwareengineeringdaily.com/2020/04/29/prefect-data...

sequoia 6y ago

> A curated list of awesome pipeline toolkits Is this "curated"? It seems like an exhaustive "dump" of toolkits.

unixhero 6y ago
What curation do you feel it lacks?
- sequoia 6y ago
  To curate a collection is to be an editor, determining what to include and what to exclude. From a curated selection of tools, I expect to see a selection of the best tools, chosen by a knowledgable curator who has evaluated the tools in some way. So for example, if you said "I want to start a library, donate any books you have at my house" that would not be a "curated" collection in my opionion. If you went through & evaluated the books, selecting only those that you'd personally recommend and discarding the rest, that would be a "curated" collection. (Strictly speaking any maintenance of a collection even just "dump everything at my door at I'll put it on the pile" could be considered "curation" but to call a list of tools "curated" suggests there's some selection going on, and there does not appear to be on this list.) EDIT: In fact, all of the definitions of "curate" here[0] start with the word "select," for example "Select, organize, and look after the items in (a collection or exhibition)." This fits my meaning. This definition[1] of "curator" requires one to merely have "care and superintendence of something," so in that sense the list is "curated" insofar as someone looks at PRs and clicks the "merge" button. The Miriam-Webster definition, however, speaks of curating a more general "something" rather than a "collection." If you "curate" a statue by cleaning & protecting it, fine, you don't need to select the statue. However, when a collection is curated, in my opinion, this necessarily implies selection, not just maintenance. 0: https://www.lexico.com/en/definition/curate https://www.lexico.com/en/definition/curate 1: https://www.merriam-webster.com/dictionary/curator https://www.merriam-webster.com/dictionary/curator