8 ms·
Airflow 2.0
- rywalker 6y agoWe've outlined some of the key new features of Airflow 2.0 here - https://www.astronomer.io/airflow https://www.astronomer.io/airflow - and it's available on Astronomer Cloud today.
- jamestimmins 6y agoAirflow 2.0 is a major release that's six months in the making. In addition to being much, much faster, this release includes HA compatibility, a new REST API, a refreshed UI, and a much simpler DAG creation experience.
- arthurcolle 6y agoCongrats on the release! Is there anywhere I can see pictures of the "refreshed UI" ?
- rywalker 6y agoThe designer talks about it in this video https://youtu.be/WJL00d1JB4Y?t=70 https://youtu.be/WJL00d1JB4Y?t=70
- arthurcolle 6y agoCool, thanks. Also on this GitHub issue - https://github.com/apache/airflow/pull/11195 https://github.com/apache/airflow/pull/11195
- jamestimmins 6y agoThanks! The feature that's gotten a lot of attention is the auto-refresh on the DAG UI, so you don't have to manually refresh the window. https://user-images.githubusercontent.com/3267/96045809-852dd000-0e40-11eb-9ff1-a3c7c14192c7.gif https://user-images.githubusercontent.com/3267/96045809-852d...
- dimberman 6y agoHi y'all! Airflow PMC here! Feel free to AMA about Airflow's new features/the roadmap going forward!
- politelemon 6y agoHey wanted to give a bit of feedback. We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and lets the Fargate containers do the heavy or complicated lifting in language of developer's choice. We'd really appreciate if the ECS Operator could be given a bit of attention: Running a task on FARGATE_SPOT containers is a cheap, convenient option, but it requires passing capacityProviderStrategy in. https://issues.apache.org/jira/browse/AIRFLOW-6604 https://issues.apache.org/jira/browse/AIRFLOW-6604 Also currently the ECSOperator only shows the output logs once the task has finished (which could take hours), it'd be better if the operator could poll the Cloudwatch logs during the run rather than wait for it to finish. --- Congratulations on the release, I'm looking forward to upgrading soon, and trying out the new features and syntax!
- dimberman 6y agoThank you for the feedback! I'm gonna pass that on to some AWS experts in the community. One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.
- hbcondo714 6y agoDitto, running Airflow on AWS ECS Fargate serverless. We did this prior to AWS announcing their Managed Workflows for Apache Airflow[1]. Do you know if and when AWS will be making Airflow 2.0 available in their managed service? [1] https://aws.amazon.com/managed-workflows-for-apache-airflow/ https://aws.amazon.com/managed-workflows-for-apache-airflow/
- politelemon 6y ago
- holdenk 6y agoCongratulations on the release :)
- dimberman 6y agoThank you Holden! :D I seriously love that you're an Airflow user since your spark talks first got me into OSS.
- mchusma 6y agoAwesome features. A great release.
- arthurcolle 6y agoI've been using Airflow for the past 6 months extensively in my projects and it is such a pleasure to use. I am excited to try out this release. Had a little trouble integrating Selenium and webDriver using BashOperators that run individual Python scripts and should probably switch to PythonOperators soon.
- dimberman 6y agoIf you're switching to python operators you should check out the TaskFlow API. You can basically build python operators with just python functions and decorators.
- arthurcolle 6y agoCan you still chain dependent tasks together with the cute ">>" syntax? I like that, makes reasoning about task orderings and co-dependent tasks pretty easy to write and think about.
- dimberman 6y agoUnfortunately that is one feature that we had to take out, but you should check out the TaskFlow API, it's a very worthwhile trade-off! Edit: Sorry I misspoke here The only thing that is no longer allowed is using a bitshift operator between a DAG and a task. task_1 >> task_2 is totally fine my_dag >> task_1 Is no longer allowed. Apologies for the miscommunication.
- shmoogy 6y agoSo if I am on 1.10.6, and upgrade, all of my existing scripts will break as >> is removed?
- dimberman 6y agoWAIT. Sorry I misspoke. The only thing that is no longer allowed is using a bitshift operator between a DAG and a task. task_1 >> task_2 is totally fine my_dag >> task_1 Is no longer allowed. Most of your DAG should be completely fine. Apologies for the miscommunication.
- superyesh 6y ago>Massive Scheduler performance improvements THIS! Mainly looking forward to the scheduler HA improvements! That has been the biggest pain in a self managed Airflow environment.
- dimberman 6y agoYeah we've been benchmarking for a while, it is VISIBLY noticeable. You're gonna love it :)
- pighive 6y agoCongratulations! I came across Airflow being used mostly for ETL space. But are there any other use cases that it is suitable for? Is it comparable to (or build) a BPMN style workflows? Thank you.
- dimberman 6y agoI would say at this point Airflow is leaning pretty heavily on being a data tool. I wouldn't recommend it for something like CI/CD for example. Do you have a use-case in mind?
- x87678r 6y agoEvery corporate job I've worked at had autosys or control-m to schedule linked jobs - not just data stuff. Would be good to replace those systems.
- contravariant 6y agoIf you can turn it into a bunch of python (or bash) scripts that should be scheduled after each other you could probably use airflow. I'm not saying you should, but it's perfectly doable. Airflow works best when you're free to restart jobs though, that's probably something to keep in mind. And Airflow is designed to spread tasks between a set of available workers, so while you can make jobs that trigger something remotely through SSH (for example) I'm still looking for a way to have a 'remote worker' that runs airflow jobs on the remote system itself rather than through an SSH connection (mostly because of a rather peculiar use-case where we'd prefer to run jobs on a remote network without two-way communication).
- x87678r 6y agoThanks, I tried airflow before and gave up because it was more complex to set up than I hoped but will try again with 2.0
- 6y ago
- Buttons840 6y agoI looked at Airflow a long time ago, and it seemed focus on running tasks based on a time-based schedule. For example, if I needed to create a daily sales report every night, Airflow would be great because I could set up a timer to kick off a reporting process every night. Then I could do things like rerun the last 5 days of reports if needed, etc. I was more interested in event-based tasks though. For example, a sale happens, and then a report about that individual sale gets generated. At the time, this seemed to require some Airflow hacks and would cause me to miss out on some of its features. Is this still the case?
- f311a 6y agoI think you need an airflow sensor.
- dimberman 6y agoA sensor would also work here. Especially with the new SmartSensor feature, sensors are basically free so you can set them up for event-based DAG executions.
- matthewbauer 6y agoWhen I looked at it, it looked like sensors were all polling based. Things like wait until another task completes or this file appears. Is there a way to make them push/event based?
- dimberman 6y agoWe didn't separate task instances from timestamps YET purely because there was already so much to release that we didn't want to add more potential for bugs/upgrade difficulties. I believe this is on the docket for 2.1 or 3.0 depending on whether it requires a breaking change (that said we plan to release much more frequently going forward so we're planning to have this feature in 2021)
- recov 6y agoAnother option besides sensors is using airflows API, which with 2.0 is much improved - http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html http://apache-airflow-docs.s3-website.eu-central-1.amazonaws...
- rscrawfo 6y agoExcited for this! I was just checking this morning to see if there was any update, but I thought it was still weeks. Nice to see it a few hours later.
- timwis 6y agoDoes anyone recommend a simpler alternative to airflow for simple multi step workflows that isolate task dependencies?
- willvarfar 6y agoOne way to interpret the 'simpler' requirement is to suggest Luigi. Its a python framework. Its way simpler if you are a programmer. Of course a lot of people think of Airflow as 'simpler' than Luigi, because Airflow is all graphical. YMMV.
- dimberman 6y agoAirflow 2.0 :)
- x87678r 6y agodagster
- morelandjs 6y agoReally happy with Prefect for smaller workflows
- eobropta 6y agoI recommend checking out Conducto. It handles dependencies well by leveraging Docker and it is very easy to get started. It can also handle super complicated pipelines that required scaled up processing by just changing one input on the command line.
- tingletech 6y agoI wonder if Amazon Managed Workflows for Apache Airflow will support this new version. https://aws.amazon.com/managed-workflows-for-apache-airflow/ https://aws.amazon.com/managed-workflows-for-apache-airflow/
- dimberman 6y agoI'm sure they will eventually, but Astronomer and (I believe) Cloud Composer have day 1 support.
- dmead 6y agoanybody have experience with this vs argo?
- igor47 6y agoWhen I last looked, Argo was nicer to run on k8s, though I still ended up writing my own system https://igor.moomers.org/building-etl-kubernetes https://igor.moomers.org/building-etl-kubernetes
- enw 6y agoGood to see a pleasant UI from an Apache project for a change.
- rectang 6y agoI wonder if there's anything different about the Airflow community as opposed to other Apache communities with regards to UI design management. ASF communities work hard to be open to contributions from any and all parties, which is important for avoiding alienation of minority factions and keeping the community from fragmenting, and also for maintaining project independence and not getting captured by a single dominant entity. As a side effect, contributions are often negotiated but rarely rejected, which exerts pressure in the direction of feature bloat: the acute needs of the contributor tend to outweigh the diffuse benefit of simplicity. Such pressure is not impossible to resist, but it usually takes dedication and and superior negotiation skills from core maintainers. A modular architecture is also very helpful for accommodating contributors without compromising usability, which is why so many successful ASF projects use plugins.
- dimberman 6y agoI'd say it's simpler than that. Airflow is a backend project built by backend engineers. Most UI people don't use Airflow or know what it is. @ryanhamilton is the first front-end dev to become a committer on the project and that JUST happened a few months ago.
- hn2017 6y agoHe's done an awesome job.. kudos to him.
- dimberman 6y agoWe're hiring multiple UI experts to rebuild the Airflow UI from scratch in react, so this is just the beginning :).
- jaikant77 6y ago
- didip 6y agoCongratulations! I am a happy user of Apache Airflow. If I may hijack this thread for a small feature request plea. I wish that the gantt chart is exposed as an API. Or, simply support alerting directly inside the gantt chart. It would make monitoring and alerting much much simpler.
- coding123 6y agoOur team has been interested in getting into Airflow on K8s, this might help push us over the scary hill.
- dimberman 6y agoOn the OSS side we have a helm chart that is heavily based on the one we use at Astronomer. That should hopefully get you started (or you can reach out to astro and someone will help out with a demo if you want help on that)
- aidenn0 6y agoI'd not heard of airflow before. Some of the other comments mention using it for ETL jobs. The webpage makes it sound like a generic job scheduler. Jenkins is also a generic job scheduler, but it is primarily used for CI/CD. It seems like there's some overlap here though; I'm interested to hear other peoples thoughts on "I have a bunch of jobs that need to be run; they aren't necessarily ETL or CI/CD; what tool is good for this, and how would I differentiate between them." It seems that mostly people use whatever they have setup, so if someone already does CI/CD through jenkins, they will add other unrelated jobs to jenkins just to avoid setting something else up. Any thoughts on if things are being left on the table here, as even generic tools can make you feel like you are "swimming up stream" if you are doing things that the rest of the community is not with them.
- pablobaz 6y agoI’ve used both extensively. Here is my personal opinion of some Pros and Cons for the types of jobs that are in the fuzzy area of overlapping functionality. Jenkins gui and interface handles lots of standalone jobs more nicely. Jenkins has much better support for parameterised jobs that can be kicked off manually. Airflow can handle dependencies between jobs in a much better way. Nicely defining and visualising dags of job really is the killer feature. Browsing task/job logs is nicer in Airflow IMO. Airflow scheduler is flaky - hopefully better in 2.0. Airflow has much more “magic” than Jenkins this is often infuriating. All that said, my preference is to move jobs to Airflow. Getting a nice gui for manually triggered jobs in airflow is/was the only large missing piece for me.
- mixmastamyk 6y ago> Getting a nice gui for manually triggered jobs… So there is a GUI, but it isn't nice. Am I reading that correctly? Or perhaps the functionality doesn't exist at all? Or can be done at the command line or API?
- shmoogy 6y agoI love airflow because it shows you the tasks in a tree view, and dag view, and you can trigger jobs manually. The UI is fine, but looks to be improved in 2.0. Main improvement I am looking forward to are the scheduler changes, I run a few jobs that trigger every minute in airflow, and it's definitely not intended for that.
- BaconJuice 6y agoCan someone ELI5 what Airflow is and why I should care? I'm seriously out of the loop on this and would love to know, thank you.
- Proven 6y agoWhy, when you're too lazy to visit the site and spend 2 minutes reading the About page?
- skibbityboop 6y agoHere here! The docs don't really make it clear what Airflow is for someone who hasn't already used or been exposed to it. Is it a job scheduler like Rundeck? Something else?
- morelandjs 6y agoIt's a job scheduler that runs directed acyclic graphs. A key feature of Airflow is you can more or less write your workflow naturally in python, and it will create the workflow graph for you. It includes a web interface for managing your workflows, a scheduler, and much more.
- webmobdev 6y agoWhat exactly is a "workflow"? And how does it benefit from "directed acyclic graphs"?
- morelandjs 6y agoLooks like a lot of options that were introduced in Prefect are now being introduced in Airflow. Obviously Prefect borrows heavily from Airflow's lineage. Seems like a little competition has been good for the space.
- Upitor 6y agoAnyone has experience with using on Windows (WSL, docker)?