10 ms·

Airflow and the Future of Data Engineering: A Q&A

201 points by scapecast 10y ago

nl 10y ago
Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and perhaps we’re more complementary than competition. From what I gather, the main maintainer of the product has left Spotify and apparently they are now using Airflow internally for [at least] some of their use cases. I do not have the full story here and would like to hear more about it. I’m thinking that many of the companies choosing Luigi today might also choose Airflow later as they develop the need for the extra set of features that Airflow offers. But there are 2 day old commits in the Luigi directory, so I don't know. I like Airflow too, but it did seems a lot more complicated the Luigi when I played with it. [1] https://github.com/spotify/luigi https://github.com/spotify/luigi
- concam 10y ago
  They've been listed as contributors on the Apache Airflow repo[0] [0] https://github.com/apache/incubator-airflow/blame/master/README.md#L145 https://github.com/apache/incubator-airflow/blame/master/REA...
- smooc 10y ago
  As far as we know Airflow is used in one of the teams (I think it maybe advertising). It does not mean the whole company has switched to Airflow, but some team decided it fitted their job better.
- ktamura 10y ago
  If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2]. It's Ansible for Workflow Management. While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems. If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure providers: you should probably have your data in multiple cloud providers as well. [1] https://www.digdag.io https://www.digdag.io [2] https://github.com/treasure-data/digdag https://github.com/treasure-data/digdag
  - ubolonton_ 10y ago
    Thanks! I skimmed through the doc. It looks very clean from a high-level view.
  - caravel 10y ago
    [auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!
    - ktamura 10y ago
      Thanks! Airflow was/has been a great source of inspiration =)
- erikbern 10y ago
  I'm one of the original Luigi authors, and used to maintain it for a while, merging PR's daily etc. I left Spotify a couple of years ago. Since then it's been maintained mostly by Arash Rouhani, who is increasingly getting busy with other things. But it's very much in active development and there are multiple pull requests merged every day. I haven't had a lot of time to check out Airflow, but it seems great. Data engineering and thinking of data processing as functional pipelines is a great paradigm, think we're going to see a lot of future development in this area. Luigi will probably evolve a lot over the next few years. Eventually I think there will be better frameworks. No idea if Airflow is a step change, I think there are still projects yet to be built that unifies everything beautifully
- caravel 10y ago
  I also hear that Uber forked Airflow early on [twice]. I'm curious to hear where they're at at this time.
glial 10y ago
+1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled. I do wish it had a REST API though.
- fuzzylearner 10y ago
  You can always expose the REST API. Its pretty easy considering they are just flask blueprints. Since, you can make your own custom plugin - You can build a lot using existing infrastructure that Airflow provides.
- papercruncher 10y ago
  I've been keeping close tabs on the project for a while now and it seems that version 1.8, which should be released in a few days, has the beginning of a rudimentary API. It also looks like more endpoints are being planned for subsequent releases
- zodiac 10y ago
  We're working on moving more functionality to a REST API - there's already some code for this in master :)
fuzzylearner 10y ago
Amazing framework with a lot of functionalities. A tool built to be extensible just what an open source software should look like :)
kozikow 10y ago
Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
- deleted 10y ago
  [deleted]
- chishaku 10y ago
  Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task. Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph. All the flexibility should be defined in whatever function, script or program you define to generate the graph.
  - concam 10y ago
    I think they are referring to "event-driven" DAGs which would be both shaped and triggered dynamically. You can accomplish this now but it feels a bit hacky and is pretty clear that it goes against the Airflow paradigm of static, slowly-changing workflows
    - kozikow 10y ago
      Yes, exactly my point.
    - smooc 10y ago
      Starting from 1.8 you will be able to trigger dags through a rest API, that is fully supported. Shaping DAGs dynamically poses a challenge to the scheduler on how to 'predict' what tasks need to run in the future. The scheduler needs to evaluate which tasks will need to run, without actually executing these tasks themselves. For Airflow in its current state that is a chicken and egg problem. For the future, I can think of allowing dynamic dags being described through the Rest API, but that is definitely further out and has not really popped up yet on the horizon.
cardosof 10y ago
I'm used to running R scripts with cron to handle some batch jobs and I'm no python dev. Would it be easy to start using Airflow?
- smooc 10y ago
  DAGS in Airflow can just be a few lines. Some understanding of the syntax of python is required. But you can start simple and add complexity as you require it.
  - classybull 10y ago
    To add to that, you can create a DAG class that's really just a wrapper that executes your R script and waits for its return code.
batmansmk 10y ago
A lot of name dropping and unprovable statements. I'm really interested by the domain and progress, but Airflow needs to be more generous in real information and less in marketing bs. Can someone share a more introductory article about what makes Airflow different from the current state of the art?
- smooc 10y ago
  Here is, a slightly outdated, article that compares several ETL workflow tools http://bytepawn.com/luigi-airflow-pinball.html http://bytepawn.com/luigi-airflow-pinball.html . Why we choose Airflow was because of the following reasons: * Scheduler that knows how to handle retries, skipped tasks, failing tasks * Great UI * Horizontal scaleable * Great community * Extensible; we could make it work in an enterprise context (kerberos, ldap etc) * No XML * Testable and debug-able workflows
- caravel 10y ago
  [author] sorry you feel that way. I understand that the section on other similar tools is controversial, especially if you work on one of those. I'm repeating what I hear being very active in this community, and answering the question that was asked to the best of my knowledge. I'm open to editing the article with better information if anyone wants to share more insight. As mentioned about Luigi, I do not have the whole story, one fact I know is that someone from Spotify gave a talk I did not attend in NYC at an Airflow meetup, and I've heard the original author had left the company. Those are provable statements, happy to debunk on the article if needed. What do you mean by [current state of the Art]?
cocoflunchy 10y ago
The actual article is here: https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data-engineering-a-q-a-266f68d956a9#.xtu264vu2 https://medium.com/the-astronomer-journey/airflow-and-the-fu...
throwaway_374 10y ago
Minor gripe - why can I not execute an entire DAG (end to end) from the UI? Also trying to execute single tasks from the UI using the "run" functionality gives a CeleryExecutor requirement error... sorry, I know this isn't the help forums but it sounds like the most trivial tasks were overlooked.
- detroitcoder 10y ago
  Actually you can but it is a bit clunky. Go to Browse > Dag Runs and select the 'Create' tab. This pulls up a form where you type in the Dag ID, enter the start time (set to now but keep in mind it is the local time of the web server), and a Run ID. Totally agree there should be both a simple Start Run and Stop Run button.
  - tedmiston 10y ago
    What I've been doing when debugging is flipping the DAG switch to off, clearing the first task in the DAG so the whole thing re-runs, then back to on.
- frugalmail 10y ago
  Azkaban has this.
sandGorgon 10y ago
MHO There seems to be quite some conceptual overlap between Airflow's DAG and Spark RDD. It seems to me that Airflow is Spark-on-a-db ... or rather Spark is Airflow-on-Hadoop. Does anyone know what the difference is ?
- gallamine 10y ago
  Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.
  - sandGorgon 10y ago
    so - celery + spark ? or just Celery Canvas ? (http://docs.celeryproject.org/en/latest/userguide/canvas.html http://docs.celeryproject.org/en/latest/userguide/canvas.htm...) P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ? For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq backend. what does Airflow do differently ?
    - smooc 10y ago
      Airflow uses Celery to horizontally scale its execution. The Airflow scheduler takes care of what tasks to run in what order, but also what to do when they fail, need to retry, don't need to run at all, backfill the past etc. Spark for Airflow is just one of the engines where a transformation of data can happen.
    - tedmiston 10y ago
      I definitely get where you're coming from. At Astronomer, we use both Airflow and Spark, though Spark is very new to me. For us, Airflow manages workflows and task dependencies but all of the actual work is done externally. Each task (operator) runs whatever dockerized command with I/O over XCom. Note that we use a custom Mesos executor instead of the Celery executor. An Airflow DAG might kick off a different Spark job based on upstream tasks.
classybull 10y ago
We're currently in a PoC phase of implementing Airflow and testing it out versus Luigi. So far, what I've liked, is that Airflow seems to be much more extensible and modular than Luigi. Getting Luigi to play nicely with our particular set of constraints was painful, and subclassing the Task was also tricky because it was way more opinionated about the structure of the class. Airflow seems way less so. There also seems to be way more right out of the gate in terms of built in task types. And the UI looks nicer. We haven't made our final determination yet, but Airflow at the current moment feels better.
- caravel 10y ago
  [Airflow author here] one of the main differences between Airflow and Luigi is the fact that in Airflow you instantiate operators to create tasks, where with Luigi you derive classes to create tasks. This means it's more natural to create tasks dynamically in Airflow. This becomes really important if want to build workflows dynamically from code [which you should sometimes!]. A very simple example of that would be an Airflow script that reads a yaml config file with a list of table names, and creates a little workflow for each table, that may do things like loading the table into a target database, perhaps apply rules from the config file around sampling, data retention, anonymisation, ... Now you have this abstraction where you can add entries to the config file to create new chunks of workflows without doing much work. It turns out there are tons of use cases for this type of approach. At Airbnb the most complex use case for this is around experimentation and A/B testing. I gave a talk at a Python meetup in SF recently talking about "Advanced data engineering patterns using Apache Airflow", which was all about dynamic pipeline generation. The ability to do that is really a game changer in data engineering and part of the motivation behind writing Airflow the way it is. I'm planning on giving this talk again, but maybe I should just record it and put it on Youtube. It's probably a better outlet than any conference/meetup...
dataops 10y ago
Data engineering is converging under the umbrella of DataOps. For those interested, there's a DataOps Summit in Boston this June https://www.dataopssummit.com/ https://www.dataopssummit.com/
- batbomb 10y ago
  This is just Data Management, a term which predates "DataOps" by more than a decade in both research and enterprise. I don't really think it needs a rebranding.
  - dataops 10y ago
    That's not really the case... There's a nice, short podcast that was just posted that defines DataOps more broadly - spend 20 minutes listening to it, and see what you think. Interested to hear your thoughts. https://thenewstack.io/delving-dataops-matters/ https://thenewstack.io/delving-dataops-matters/
jcalabro 10y ago
[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases. In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, orchestrate, or analyze your data using our built-in Modules (blocks) or you can easily write your own. You can then view your resulting data all within the webapp. Reading the comments, it seems like Composable supports a lot of the things people are asking for here that Airflow is lacking. Maybe check us out and let us know what you think! For more information: Composable Site - https://composableanalytics.com/ https://composableanalytics.com/ Try it yourself - https://cloud.composableanalytics.com/ https://cloud.composableanalytics.com/ Composable's Blog - http://blog.composable.ai/ http://blog.composable.ai/
- j_s 10y ago
  Is any aspect of your product open source? Doesn't look like it; this makes it a bit of an apples-vs-apple-pie comparison.
- batbomb 10y ago
  It sounds like you are more of a Beam/dataflow competitor than an airflow/workflow competitor.
- caravel 10y ago
  [author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software: https://medium.freecodecamp.com/the-rise-of-the-data-engineer-91be18f1e603 https://medium.freecodecamp.com/the-rise-of-the-data-enginee... I'd also argue for open source over proprietary, mostly to allow for a framework that is "hackable" and extensible by nature. You can also count on the community to build a lot of the operators & hooks you'll need (Airflow terms).
  - larsf 10y ago
gregn610 10y ago
How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?
- colinwilson 10y ago
  I would say it can be used at it's simplest as a replacement for cron. It supports running programs on a schedule and you can set concurrency rules, SLAs, and triggers around even single commands or programs. You also get nice graphs of task run time and email alerts if jobs take longer than the SLA you've set.
- gdulli 10y ago
  Airflow is more about scaling your number of processes, not the size of your data. It's relevant if you want to have a higher-level system for managing processes independent of what those processes are.
- amalag 10y ago
  I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.
  - gregn610 10y ago
    Talend makes my teeth grind. I don't understand why an ETL tool uses a strongly typed language for a foundation. The number of fun productive hours I've spent swapping chars to varchars, int to decimal & vice versa. In 2017 computers can read a registration plate from a blurry photograph and spot a criminal in a stadium, but a user puts apostrophe in a CSV file and schmoo leaks everywhere.
    - vira 10y ago
      Strong typing helps. Keep in mind enterprise ETL tools are designed to move data from oltp to olap databases, which are often strongly typed as well.
deleted 10y ago
[deleted]
tedmiston 10y ago
Hey HN, I co-wrote the article with Maxime. A little late to the party here but happy to answer any questions you might have about it. I'll send him the link as well. By the way, thank you to Maxime for sharing his thoughts, and the Astronomer team for contributing great questions.
- caravel 10y ago
  Maxime reporting for duty here, I'll go through the thread and answer questions / add comments.
botswana99 10y ago
[Bias Alert: I'm Head Chef of DataKitchen]. Our perspective is that the DAG abstraction should not apply only to data engineering, but the whole analytic process of data engineering, data science, and data visualization. Analytic teams love to work with their favorite tools -- Python, SQL, ETL, Jupyter, R, Tableau, Alteryx, etc. The question is how do you get those diverse teams and tools to work together to deliver fast, with high quality, and reusable components? We've identified seven steps taken from DevOps, CI, Agile and Lean Manufacturing (https://www.datakitchen.io/platform.html#sevensteps https://www.datakitchen.io/platform.html#sevensteps) that you can start to apply today. We also created a 'DataOps' platform that incorporates those principles into a software: https://www.datakitchen.io https://www.datakitchen.io. The challenge is that there are many separate DAGs (and code and configuration) involved in producing complete production analytics embedded in each of the tools the team has selected. So what is needed is a “DAG of DAGs” that encompasses the whole analytic tool chain.
- caravel 10y ago
  [Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform. At Airbnb Airflow is far from being limited to data engineering. All the batch scheduling goes through Airflow and many team (data science, analysts, data infra, ML infra, engineering as a whole, ...) uses Airflow in all sorts of ways. Airflow has a solid story in terms of reusable components, from extendable abstractions (operator, hooks, executors, macros, ...) all the way to computation frameworks.
LevonK 10y ago
Disney Studios uses Azkaban because it's language agnostic, and it believes that in the data space there is a huge advantage to static (and strong, but that's beside the point in this argument) typing in the data space. The language agnostic aspect means that non software engineers can also use the orchestration platform for runbook automation.
jaz46 10y ago
One of the important paradigms that I think Luigi and Airflow miss is that they treat pipelines as a DAG of tasks, when it really should be thought of as a DAG of Data. It's a subtle difference, but has huge impacts when you're trying to dynamically scale tasks based on cluster resources and track data lineage throughout your system.(Disclosure: I'm the founder of Pachyderm[0], a containerized data pipeline framework where we version control data in this way). Check out Samuel Lampa's post[1] about dynamically scaling data pipelines for more details. [0] github.com/pachyderm/pachyderm [1] http://bionics.it/posts/dynamic-workflow-scheduling http://bionics.it/posts/dynamic-workflow-scheduling
- caravel 10y ago
  [Airflow author] The task is centric to the workflow engine. It's an important entity, and it's complementary to the data lineage graph (not necessarily a DAG btw). At Airbnb we have another important tool (not open source at the moment) that is a UI and search engine to understand all of the "data objects" and how they relate. It includes datasets, tables, charts, dashboards and tasks. The edges are usage, attribution, sources, ... This tool shows [amongst other things] data lineage and is complementary to Airflow.
  - jaz46 10y ago
    Does the other tool you're talking about that works with Airflow allow you to scale your Airflow tasks based on, for example, the amount of new input data that needs to be processed? That was one of the major challenges we see in bioinformatics workloads. Sometimes you have a few new samples to run and other times there are thousands -- so your task scheduler, although it is centric, needs to have an understanding of the data too.
    - caravel 10y ago
      At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it triggers. It's possible to use a feature called XCom as a message bus between tasks, but would typically direct people in the direction of having stateless, idempotent, "contained" units of work and avoid cross task communication as much as possible. https://airflow.incubator.apache.org/concepts.html#xcoms https://airflow.incubator.apache.org/concepts.html#xcoms For your case [which I have little input on] I think singleton DAGs described in another post on this page may work.
mtrn 10y ago
Luigi is nice, because it is really simple to get started and it gradually allows you to do more complex things like (custom) parameter types, custom targets, enhanced super classes and dynamic dependencies, event hooks, task history and more. One thing that I missed a bit was automatic task output naming based on the parameters of a task, so I wrote a thin wrapper for that [1]. This helps, but mostly for smaller deployments. Airflow and luigi seemed to me like two side of the same thing: fixed graphs vs data flow. One fixates the DAG, the other puts more emphasis on composition. That said, I am excited about the data processing tools to come - I believe this is an exciting space and choosing or writing the right tool can make a real difference between a messy data landscape and an agile part of business and business development. [1] https://github.com/miku/gluish https://github.com/miku/gluish
- jaz46 10y ago
  Definitely agree that that is one of the great points with Luigi. Airflow's UI of course blows everything out of the water IMO. As for organizing your data, my personal and very biased opinion is that version control semantics similar to Git [0] are a pretty good way to help tame the complexity of ever-changing data sets. We already version code, but with versioned data too, now everything becomes completely reproducible. [0] Our Data Science Bill of Rights: http://www.pachyderm.io/dsbor.html http://www.pachyderm.io/dsbor.html
  - mtrn 10y ago
    The git angle would be a huge step forward. What I found is that reproducability is not always on people's mind when they designing such systems, whereas I believe it's one of the most important properties. Pachyderm is on my TODO list for a while, so thanks for reminding me, I'll try to implement something real with it soon.
vicaya 10y ago
Who has used both Airflow and Spinnaker? Feedbacks?
somewhatoff 10y ago
How would you see Airflow in relation to Apache Beam / GC Dataflow?
- caravel 10y ago
  [author] Airflow is not a data flow engine, though you can use it to do some of that, but we typically defer on doing data transformations using/coordinating external engines (Spark, Hive, Cascading, Sqoop, PIG, ...). We operate at a higher level: orchestration. If we were to start using Apache Beam at Airbnb (and we very well may soon!), we' use Airflow to schedule and trigger batch beam jobs alongside the rest of our other jobs.
  - somewhatoff 10y ago
    Thanks, that's really interesting. The usage of 'pipeline' to describe both sorts of system made me think there was a lot of overlap, but I'm understanding now how they are complementary.

201 points by scapecast 10y ago

nl 10y ago

Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and perhaps we’re more complementary than competition. From what I gather, the main maintainer of the product has left Spotify and apparently they are now using Airflow internally for [at least] some of their use cases. I do not have the full story here and would like to hear more about it. I’m thinking that many of the companies choosing Luigi today might also choose Airflow later as they develop the need for the extra set of features that Airflow offers. But there are 2 day old commits in the Luigi directory, so I don't know. I like Airflow too, but it did seems a lot more complicated the Luigi when I played with it. [1] https://github.com/spotify/luigi https://github.com/spotify/luigi

concam 10y ago
They've been listed as contributors on the Apache Airflow repo[0] [0] https://github.com/apache/incubator-airflow/blame/master/README.md#L145 https://github.com/apache/incubator-airflow/blame/master/REA...
smooc 10y ago
As far as we know Airflow is used in one of the teams (I think it maybe advertising). It does not mean the whole company has switched to Airflow, but some team decided it fitted their job better.
ktamura 10y ago
If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2]. It's Ansible for Workflow Management. While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems. If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure providers: you should probably have your data in multiple cloud providers as well. [1] https://www.digdag.io https://www.digdag.io [2] https://github.com/treasure-data/digdag https://github.com/treasure-data/digdag
- ubolonton_ 10y ago
  Thanks! I skimmed through the doc. It looks very clean from a high-level view.
- caravel 10y ago
  [auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!
  - ktamura 10y ago
    Thanks! Airflow was/has been a great source of inspiration =)
erikbern 10y ago
I'm one of the original Luigi authors, and used to maintain it for a while, merging PR's daily etc. I left Spotify a couple of years ago. Since then it's been maintained mostly by Arash Rouhani, who is increasingly getting busy with other things. But it's very much in active development and there are multiple pull requests merged every day. I haven't had a lot of time to check out Airflow, but it seems great. Data engineering and thinking of data processing as functional pipelines is a great paradigm, think we're going to see a lot of future development in this area. Luigi will probably evolve a lot over the next few years. Eventually I think there will be better frameworks. No idea if Airflow is a step change, I think there are still projects yet to be built that unifies everything beautifully
caravel 10y ago
I also hear that Uber forked Airflow early on [twice]. I'm curious to hear where they're at at this time.

glial 10y ago

+1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled. I do wish it had a REST API though.

fuzzylearner 10y ago
You can always expose the REST API. Its pretty easy considering they are just flask blueprints. Since, you can make your own custom plugin - You can build a lot using existing infrastructure that Airflow provides.
papercruncher 10y ago
I've been keeping close tabs on the project for a while now and it seems that version 1.8, which should be released in a few days, has the beginning of a rudimentary API. It also looks like more endpoints are being planned for subsequent releases
zodiac 10y ago
We're working on moving more functionality to a REST API - there's already some code for this in master :)

fuzzylearner 10y ago

Amazing framework with a lot of functionalities. A tool built to be extensible just what an open source software should look like :)

kozikow 10y ago

Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.

deleted 10y ago
[deleted]
chishaku 10y ago
Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task. Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph. All the flexibility should be defined in whatever function, script or program you define to generate the graph.
- concam 10y ago
  I think they are referring to "event-driven" DAGs which would be both shaped and triggered dynamically. You can accomplish this now but it feels a bit hacky and is pretty clear that it goes against the Airflow paradigm of static, slowly-changing workflows
  - kozikow 10y ago
    Yes, exactly my point.
  - smooc 10y ago
    Starting from 1.8 you will be able to trigger dags through a rest API, that is fully supported. Shaping DAGs dynamically poses a challenge to the scheduler on how to 'predict' what tasks need to run in the future. The scheduler needs to evaluate which tasks will need to run, without actually executing these tasks themselves. For Airflow in its current state that is a chicken and egg problem. For the future, I can think of allowing dynamic dags being described through the Rest API, but that is definitely further out and has not really popped up yet on the horizon.

cardosof 10y ago

I'm used to running R scripts with cron to handle some batch jobs and I'm no python dev. Would it be easy to start using Airflow?

smooc 10y ago
DAGS in Airflow can just be a few lines. Some understanding of the syntax of python is required. But you can start simple and add complexity as you require it.
- classybull 10y ago
  To add to that, you can create a DAG class that's really just a wrapper that executes your R script and waits for its return code.

batmansmk 10y ago

A lot of name dropping and unprovable statements. I'm really interested by the domain and progress, but Airflow needs to be more generous in real information and less in marketing bs. Can someone share a more introductory article about what makes Airflow different from the current state of the art?

smooc 10y ago
Here is, a slightly outdated, article that compares several ETL workflow tools http://bytepawn.com/luigi-airflow-pinball.html http://bytepawn.com/luigi-airflow-pinball.html . Why we choose Airflow was because of the following reasons: * Scheduler that knows how to handle retries, skipped tasks, failing tasks * Great UI * Horizontal scaleable * Great community * Extensible; we could make it work in an enterprise context (kerberos, ldap etc) * No XML * Testable and debug-able workflows
caravel 10y ago
[author] sorry you feel that way. I understand that the section on other similar tools is controversial, especially if you work on one of those. I'm repeating what I hear being very active in this community, and answering the question that was asked to the best of my knowledge. I'm open to editing the article with better information if anyone wants to share more insight. As mentioned about Luigi, I do not have the whole story, one fact I know is that someone from Spotify gave a talk I did not attend in NYC at an Airflow meetup, and I've heard the original author had left the company. Those are provable statements, happy to debunk on the article if needed. What do you mean by [current state of the Art]?

cocoflunchy 10y ago

The actual article is here: https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data-engineering-a-q-a-266f68d956a9#.xtu264vu2 https://medium.com/the-astronomer-journey/airflow-and-the-fu...

throwaway_374 10y ago

Minor gripe - why can I not execute an entire DAG (end to end) from the UI? Also trying to execute single tasks from the UI using the "run" functionality gives a CeleryExecutor requirement error... sorry, I know this isn't the help forums but it sounds like the most trivial tasks were overlooked.

detroitcoder 10y ago
Actually you can but it is a bit clunky. Go to Browse > Dag Runs and select the 'Create' tab. This pulls up a form where you type in the Dag ID, enter the start time (set to now but keep in mind it is the local time of the web server), and a Run ID. Totally agree there should be both a simple Start Run and Stop Run button.
- tedmiston 10y ago
  What I've been doing when debugging is flipping the DAG switch to off, clearing the first task in the DAG so the whole thing re-runs, then back to on.
frugalmail 10y ago
Azkaban has this.

sandGorgon 10y ago

MHO There seems to be quite some conceptual overlap between Airflow's DAG and Spark RDD. It seems to me that Airflow is Spark-on-a-db ... or rather Spark is Airflow-on-Hadoop. Does anyone know what the difference is ?

gallamine 10y ago
Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.
- sandGorgon 10y ago
  so - celery + spark ? or just Celery Canvas ? (http://docs.celeryproject.org/en/latest/userguide/canvas.html http://docs.celeryproject.org/en/latest/userguide/canvas.htm...) P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ? For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq backend. what does Airflow do differently ?
  - smooc 10y ago
    Airflow uses Celery to horizontally scale its execution. The Airflow scheduler takes care of what tasks to run in what order, but also what to do when they fail, need to retry, don't need to run at all, backfill the past etc. Spark for Airflow is just one of the engines where a transformation of data can happen.
  - tedmiston 10y ago
    I definitely get where you're coming from. At Astronomer, we use both Airflow and Spark, though Spark is very new to me. For us, Airflow manages workflows and task dependencies but all of the actual work is done externally. Each task (operator) runs whatever dockerized command with I/O over XCom. Note that we use a custom Mesos executor instead of the Celery executor. An Airflow DAG might kick off a different Spark job based on upstream tasks.

classybull 10y ago

We're currently in a PoC phase of implementing Airflow and testing it out versus Luigi. So far, what I've liked, is that Airflow seems to be much more extensible and modular than Luigi. Getting Luigi to play nicely with our particular set of constraints was painful, and subclassing the Task was also tricky because it was way more opinionated about the structure of the class. Airflow seems way less so. There also seems to be way more right out of the gate in terms of built in task types. And the UI looks nicer. We haven't made our final determination yet, but Airflow at the current moment feels better.

caravel 10y ago
[Airflow author here] one of the main differences between Airflow and Luigi is the fact that in Airflow you instantiate operators to create tasks, where with Luigi you derive classes to create tasks. This means it's more natural to create tasks dynamically in Airflow. This becomes really important if want to build workflows dynamically from code [which you should sometimes!]. A very simple example of that would be an Airflow script that reads a yaml config file with a list of table names, and creates a little workflow for each table, that may do things like loading the table into a target database, perhaps apply rules from the config file around sampling, data retention, anonymisation, ... Now you have this abstraction where you can add entries to the config file to create new chunks of workflows without doing much work. It turns out there are tons of use cases for this type of approach. At Airbnb the most complex use case for this is around experimentation and A/B testing. I gave a talk at a Python meetup in SF recently talking about "Advanced data engineering patterns using Apache Airflow", which was all about dynamic pipeline generation. The ability to do that is really a game changer in data engineering and part of the motivation behind writing Airflow the way it is. I'm planning on giving this talk again, but maybe I should just record it and put it on Youtube. It's probably a better outlet than any conference/meetup...

dataops 10y ago

Data engineering is converging under the umbrella of DataOps. For those interested, there's a DataOps Summit in Boston this June https://www.dataopssummit.com/ https://www.dataopssummit.com/

batbomb 10y ago
This is just Data Management, a term which predates "DataOps" by more than a decade in both research and enterprise. I don't really think it needs a rebranding.
- dataops 10y ago
  That's not really the case... There's a nice, short podcast that was just posted that defines DataOps more broadly - spend 20 minutes listening to it, and see what you think. Interested to hear your thoughts. https://thenewstack.io/delving-dataops-matters/ https://thenewstack.io/delving-dataops-matters/

jcalabro 10y ago

[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases. In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, orchestrate, or analyze your data using our built-in Modules (blocks) or you can easily write your own. You can then view your resulting data all within the webapp. Reading the comments, it seems like Composable supports a lot of the things people are asking for here that Airflow is lacking. Maybe check us out and let us know what you think! For more information: Composable Site - https://composableanalytics.com/ https://composableanalytics.com/ Try it yourself - https://cloud.composableanalytics.com/ https://cloud.composableanalytics.com/ Composable's Blog - http://blog.composable.ai/ http://blog.composable.ai/

j_s 10y ago
Is any aspect of your product open source? Doesn't look like it; this makes it a bit of an apples-vs-apple-pie comparison.
batbomb 10y ago
It sounds like you are more of a Beam/dataflow competitor than an airflow/workflow competitor.
caravel 10y ago
[author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software: https://medium.freecodecamp.com/the-rise-of-the-data-engineer-91be18f1e603 https://medium.freecodecamp.com/the-rise-of-the-data-enginee... I'd also argue for open source over proprietary, mostly to allow for a framework that is "hackable" and extensible by nature. You can also count on the community to build a lot of the operators & hooks you'll need (Airflow terms).
- larsf 10y ago

gregn610 10y ago

How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?

colinwilson 10y ago
I would say it can be used at it's simplest as a replacement for cron. It supports running programs on a schedule and you can set concurrency rules, SLAs, and triggers around even single commands or programs. You also get nice graphs of task run time and email alerts if jobs take longer than the SLA you've set.
gdulli 10y ago
Airflow is more about scaling your number of processes, not the size of your data. It's relevant if you want to have a higher-level system for managing processes independent of what those processes are.
amalag 10y ago
I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.
- gregn610 10y ago
  Talend makes my teeth grind. I don't understand why an ETL tool uses a strongly typed language for a foundation. The number of fun productive hours I've spent swapping chars to varchars, int to decimal & vice versa. In 2017 computers can read a registration plate from a blurry photograph and spot a criminal in a stadium, but a user puts apostrophe in a CSV file and schmoo leaks everywhere.
  - vira 10y ago
    Strong typing helps. Keep in mind enterprise ETL tools are designed to move data from oltp to olap databases, which are often strongly typed as well.

deleted 10y ago

[deleted]

tedmiston 10y ago

Hey HN, I co-wrote the article with Maxime. A little late to the party here but happy to answer any questions you might have about it. I'll send him the link as well. By the way, thank you to Maxime for sharing his thoughts, and the Astronomer team for contributing great questions.

caravel 10y ago
Maxime reporting for duty here, I'll go through the thread and answer questions / add comments.

botswana99 10y ago

[Bias Alert: I'm Head Chef of DataKitchen]. Our perspective is that the DAG abstraction should not apply only to data engineering, but the whole analytic process of data engineering, data science, and data visualization. Analytic teams love to work with their favorite tools -- Python, SQL, ETL, Jupyter, R, Tableau, Alteryx, etc. The question is how do you get those diverse teams and tools to work together to deliver fast, with high quality, and reusable components? We've identified seven steps taken from DevOps, CI, Agile and Lean Manufacturing (https://www.datakitchen.io/platform.html#sevensteps https://www.datakitchen.io/platform.html#sevensteps) that you can start to apply today. We also created a 'DataOps' platform that incorporates those principles into a software: https://www.datakitchen.io https://www.datakitchen.io. The challenge is that there are many separate DAGs (and code and configuration) involved in producing complete production analytics embedded in each of the tools the team has selected. So what is needed is a “DAG of DAGs” that encompasses the whole analytic tool chain.

caravel 10y ago
[Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform. At Airbnb Airflow is far from being limited to data engineering. All the batch scheduling goes through Airflow and many team (data science, analysts, data infra, ML infra, engineering as a whole, ...) uses Airflow in all sorts of ways. Airflow has a solid story in terms of reusable components, from extendable abstractions (operator, hooks, executors, macros, ...) all the way to computation frameworks.

LevonK 10y ago

Disney Studios uses Azkaban because it's language agnostic, and it believes that in the data space there is a huge advantage to static (and strong, but that's beside the point in this argument) typing in the data space. The language agnostic aspect means that non software engineers can also use the orchestration platform for runbook automation.

jaz46 10y ago

One of the important paradigms that I think Luigi and Airflow miss is that they treat pipelines as a DAG of tasks, when it really should be thought of as a DAG of Data. It's a subtle difference, but has huge impacts when you're trying to dynamically scale tasks based on cluster resources and track data lineage throughout your system.(Disclosure: I'm the founder of Pachyderm[0], a containerized data pipeline framework where we version control data in this way). Check out Samuel Lampa's post[1] about dynamically scaling data pipelines for more details. [0] github.com/pachyderm/pachyderm [1] http://bionics.it/posts/dynamic-workflow-scheduling http://bionics.it/posts/dynamic-workflow-scheduling

caravel 10y ago
[Airflow author] The task is centric to the workflow engine. It's an important entity, and it's complementary to the data lineage graph (not necessarily a DAG btw). At Airbnb we have another important tool (not open source at the moment) that is a UI and search engine to understand all of the "data objects" and how they relate. It includes datasets, tables, charts, dashboards and tasks. The edges are usage, attribution, sources, ... This tool shows [amongst other things] data lineage and is complementary to Airflow.
- jaz46 10y ago
  Does the other tool you're talking about that works with Airflow allow you to scale your Airflow tasks based on, for example, the amount of new input data that needs to be processed? That was one of the major challenges we see in bioinformatics workloads. Sometimes you have a few new samples to run and other times there are thousands -- so your task scheduler, although it is centric, needs to have an understanding of the data too.
  - caravel 10y ago
    At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it triggers. It's possible to use a feature called XCom as a message bus between tasks, but would typically direct people in the direction of having stateless, idempotent, "contained" units of work and avoid cross task communication as much as possible. https://airflow.incubator.apache.org/concepts.html#xcoms https://airflow.incubator.apache.org/concepts.html#xcoms For your case [which I have little input on] I think singleton DAGs described in another post on this page may work.

mtrn 10y ago

Luigi is nice, because it is really simple to get started and it gradually allows you to do more complex things like (custom) parameter types, custom targets, enhanced super classes and dynamic dependencies, event hooks, task history and more. One thing that I missed a bit was automatic task output naming based on the parameters of a task, so I wrote a thin wrapper for that [1]. This helps, but mostly for smaller deployments. Airflow and luigi seemed to me like two side of the same thing: fixed graphs vs data flow. One fixates the DAG, the other puts more emphasis on composition. That said, I am excited about the data processing tools to come - I believe this is an exciting space and choosing or writing the right tool can make a real difference between a messy data landscape and an agile part of business and business development. [1] https://github.com/miku/gluish https://github.com/miku/gluish

jaz46 10y ago
Definitely agree that that is one of the great points with Luigi. Airflow's UI of course blows everything out of the water IMO. As for organizing your data, my personal and very biased opinion is that version control semantics similar to Git [0] are a pretty good way to help tame the complexity of ever-changing data sets. We already version code, but with versioned data too, now everything becomes completely reproducible. [0] Our Data Science Bill of Rights: http://www.pachyderm.io/dsbor.html http://www.pachyderm.io/dsbor.html
- mtrn 10y ago
  The git angle would be a huge step forward. What I found is that reproducability is not always on people's mind when they designing such systems, whereas I believe it's one of the most important properties. Pachyderm is on my TODO list for a while, so thanks for reminding me, I'll try to implement something real with it soon.

vicaya 10y ago

Who has used both Airflow and Spinnaker? Feedbacks?

somewhatoff 10y ago

How would you see Airflow in relation to Apache Beam / GC Dataflow?

caravel 10y ago
[author] Airflow is not a data flow engine, though you can use it to do some of that, but we typically defer on doing data transformations using/coordinating external engines (Spark, Hive, Cascading, Sqoop, PIG, ...). We operate at a higher level: orchestration. If we were to start using Apache Beam at Airbnb (and we very well may soon!), we' use Airflow to schedule and trigger batch beam jobs alongside the rest of our other jobs.
- somewhatoff 10y ago
  Thanks, that's really interesting. The usage of 'pipeline' to describe both sorts of system made me think there was a lot of overlap, but I'm understanding now how they are complementary.