6 ms·

Airflow brought one of the best tools with nice UI for running pipelines back in 2014-2016. But now days engineers should be aware about easier to use options a

Airflow brought one of the best tools with nice UI for running pipelines back in 2014-2016. But now days engineers should be aware about easier to use options and don’t choose Airflow blindly as default choice. IMHO for 80-90% of cases orchestration system should not use code at all - it should be DAGs as a config code. Airflow is popular and teams keep choosing it for building simple DAGs and incurring avoidable otherwise Airflow maintenance costs. Databricks Orchestration pipeline, AWS Step Functions - good examples of DAGs as a configuration.

nomilk 4y ago
> teams keep choosing it for building simple DAGs I am part of one such team. We were using Windows Task Scheduler on a windows VM to run jobs, we figured it would be a nice idea to (dramatically) modernise and move to airflow, but we grossly underestimated the complexity, learning curve, and surrounding tools it requires. In the end we (data science team) didn't get a single production task up and running. The data engineers had much more success with it thought, probably because they dedicated much more time to it. Will look forward to trying AWS Step Functions.
- commandlinefan 4y ago
  I tried installing Airflow locally to just play around with it and make sense of what it's good for and finally gave up after a few days - the install alone is insanely complicated, with lots of tricky hidden dependencies.
  - always_left 4y ago
    Did you try installing with docker? You would just download docker, `docker-compose up --build` and you'll be good to go locally (usually)
    - idleprocess 4y ago
      I can second this. We were up-and-running with Docker on our dev machines in just a few minutes. A native installation involves substantially more setup (Python, databases, Redis and/or Rabbit, etc.). The published docker-compose file will handle all of that for you. We have a very small data engineering team and have been able to move very quickly with Docker and AWS ECS (for orchestrating containers in test and prod environments).
  - flowair 4y ago
    download the astronomer CLI and it will spin up a local dockerized instance of airflow in about 1 minute tops https://docs.astronomer.io/astro/install-cli https://docs.astronomer.io/astro/install-cli
carschno 4y ago
Do you have more examples for better tools, ideally open source (unlike AWS Step functions)?
- gadflyinyoureye 4y ago
  Flowable is a BPNM system. You can do a lot of async calls with it. https://www.flowable.com/open-source https://www.flowable.com/open-source We use it for a complex pricing process that invokes 30-40 micro services that can take up to minutes per step.
- qw 4y ago
  Kamelets and the Karavan UI in combination with k8s and Knative for "serverless" integrations looks interesting.
- antupis 4y ago
  We are using prefect+dbt and I like it. Altought they are doing huge rewrite at the moment.
- jonpon 4y ago
  We at magniv.io are building an alternative. Our core is open source https://github.com/MagnivOrg/magniv-core https://github.com/MagnivOrg/magniv-core We can set you up with our hosted if you would like to poke around!
deleted 4y ago
[deleted]
avemg 4y ago
I've used AWS Step Functions extensively over the past several years and give me code every day of the week over the Stepfunctions json config. Once you get beyond a few simple steps, it gets very hard to look at the config and understand what's going on with it. Especially true when you haven't look at the config in awhile. The DAG visualizer definitely helps, but as soon as things get beyond the trivial I long for a different tool.
- coolsunglasses 4y ago
  I was until a week or two ago part of a team that build datasets with extensive dependencies (thus, complicated DAGs) v1 of the system built before I joined was Step Functions and the like. It gets hairy just as you say. v2 I built and designed with the lead data engineer, we called it Coriaria originally. We're hoping/planning to open source it eventually, although it's a little wrapped around our company's internal needs & systems. It chooses neither "config" strictly speaking nor "code" for the DAG, instead the primary representation/state is all in the PostgreSQL database which tracks the dataset dependencies and how each dataset is built. It's a DAG in PostgreSQL as well. To make dataset creation and management easier, I also wrote a custom Terraform provider for Coriaria. This made migrating datasets into the new system dramatically faster. The provider is really nice, supports `terraform import` and all that. Currently we have it setup so that there are separate roles/accounts that can modify an existing dataset, but reading state only requires authentication, not authorization. This enables one team to depend on another team's dataset as an upstream data source for their datasets without granting permission to modify it or create a potentially stale copy of the dataset. Terraform's internal DAG representation of the resource dependencies is leveraged because "parent_datasets" references the upstream datasets directly, including the ones we don't build. We're able to depend on datasets we don't build ourselves because the system has support for Glue catalog backends to track and register partition availability. Currently, it builds most of the datasets using AWS Athena & S3, however this is abstracted over a single step function. There's no DAG of step functions, it's just a convenient wrapper for the Athena query execution. The system also explicitly understands dataset builds and validations as separate steps. The dashboard makes it easy to trace the DAG and see which datasets are blocking a dataset build. We're adding more integrations to it soon so that other ways of kicking off dataset builds and validations are available. If people are interested in this I can begin lobbying for open sourcing the system. My colleague wanted to open source it as well. All else fails, I'll rebuild it from scratch because I don't like the existing solutions for managing datasets. We've been calling it a data-flow orchestration system or ETL orchestration system, not sure what would be most meaningful to people. I think the main caveat to this system is that I'm not sure how much use it'd be for streaming data pipelines, but it could manage the discretization of streaming into validated partitions wherever streamed data is sunk into. Our operating assumptions are that you want validated datasets to drive business decisions, not raw event data streamed in from Kafka. Making sure the right data is located in each daily (or hourly) partition is part of that validation.

nomilk 4y ago

> teams keep choosing it for building simple DAGs I am part of one such team. We were using Windows Task Scheduler on a windows VM to run jobs, we figured it would be a nice idea to (dramatically) modernise and move to airflow, but we grossly underestimated the complexity, learning curve, and surrounding tools it requires. In the end we (data science team) didn't get a single production task up and running. The data engineers had much more success with it thought, probably because they dedicated much more time to it. Will look forward to trying AWS Step Functions.

commandlinefan 4y ago
I tried installing Airflow locally to just play around with it and make sense of what it's good for and finally gave up after a few days - the install alone is insanely complicated, with lots of tricky hidden dependencies.
- always_left 4y ago
  Did you try installing with docker? You would just download docker, `docker-compose up --build` and you'll be good to go locally (usually)
  - idleprocess 4y ago
    I can second this. We were up-and-running with Docker on our dev machines in just a few minutes. A native installation involves substantially more setup (Python, databases, Redis and/or Rabbit, etc.). The published docker-compose file will handle all of that for you. We have a very small data engineering team and have been able to move very quickly with Docker and AWS ECS (for orchestrating containers in test and prod environments).
- flowair 4y ago
  download the astronomer CLI and it will spin up a local dockerized instance of airflow in about 1 minute tops https://docs.astronomer.io/astro/install-cli https://docs.astronomer.io/astro/install-cli

carschno 4y ago

Do you have more examples for better tools, ideally open source (unlike AWS Step functions)?

gadflyinyoureye 4y ago
Flowable is a BPNM system. You can do a lot of async calls with it. https://www.flowable.com/open-source https://www.flowable.com/open-source We use it for a complex pricing process that invokes 30-40 micro services that can take up to minutes per step.
qw 4y ago
Kamelets and the Karavan UI in combination with k8s and Knative for "serverless" integrations looks interesting.
antupis 4y ago
We are using prefect+dbt and I like it. Altought they are doing huge rewrite at the moment.
jonpon 4y ago
We at magniv.io are building an alternative. Our core is open source https://github.com/MagnivOrg/magniv-core https://github.com/MagnivOrg/magniv-core We can set you up with our hosted if you would like to poke around!

deleted 4y ago

[deleted]

avemg 4y ago

I've used AWS Step Functions extensively over the past several years and give me code every day of the week over the Stepfunctions json config. Once you get beyond a few simple steps, it gets very hard to look at the config and understand what's going on with it. Especially true when you haven't look at the config in awhile. The DAG visualizer definitely helps, but as soon as things get beyond the trivial I long for a different tool.

coolsunglasses 4y ago
I was until a week or two ago part of a team that build datasets with extensive dependencies (thus, complicated DAGs) v1 of the system built before I joined was Step Functions and the like. It gets hairy just as you say. v2 I built and designed with the lead data engineer, we called it Coriaria originally. We're hoping/planning to open source it eventually, although it's a little wrapped around our company's internal needs & systems. It chooses neither "config" strictly speaking nor "code" for the DAG, instead the primary representation/state is all in the PostgreSQL database which tracks the dataset dependencies and how each dataset is built. It's a DAG in PostgreSQL as well. To make dataset creation and management easier, I also wrote a custom Terraform provider for Coriaria. This made migrating datasets into the new system dramatically faster. The provider is really nice, supports `terraform import` and all that. Currently we have it setup so that there are separate roles/accounts that can modify an existing dataset, but reading state only requires authentication, not authorization. This enables one team to depend on another team's dataset as an upstream data source for their datasets without granting permission to modify it or create a potentially stale copy of the dataset. Terraform's internal DAG representation of the resource dependencies is leveraged because "parent_datasets" references the upstream datasets directly, including the ones we don't build. We're able to depend on datasets we don't build ourselves because the system has support for Glue catalog backends to track and register partition availability. Currently, it builds most of the datasets using AWS Athena & S3, however this is abstracted over a single step function. There's no DAG of step functions, it's just a convenient wrapper for the Athena query execution. The system also explicitly understands dataset builds and validations as separate steps. The dashboard makes it easy to trace the DAG and see which datasets are blocking a dataset build. We're adding more integrations to it soon so that other ways of kicking off dataset builds and validations are available. If people are interested in this I can begin lobbying for open sourcing the system. My colleague wanted to open source it as well. All else fails, I'll rebuild it from scratch because I don't like the existing solutions for managing datasets. We've been calling it a data-flow orchestration system or ETL orchestration system, not sure what would be most meaningful to people. I think the main caveat to this system is that I'm not sure how much use it'd be for streaming data pipelines, but it could manage the discretization of streaming into validated partitions wherever streamed data is sunk into. Our operating assumptions are that you want validated datasets to drive business decisions, not raw event data streamed in from Kafka. Making sure the right data is located in each daily (or hourly) partition is part of that validation.