4 ms·
I've spent the past month+ setting airflow up. To be honest, I don't like it for a lot of reasons: 1) it's not cloud native in the sense that running this on e
by jillesvangurp 6y ago
I've spent the past month+ setting airflow up. To be honest, I don't like it for a lot of reasons:
1) it's not cloud native in the sense that running this on e.g. AWS is an easy and well trodden path. Cloud is left as an exercise to the reader of the documentation and at best vaguely hinted at as a possibility. This is weird because that kind of is the whole point of this product. Sure, it has lots of things that are highly useful in the cloud (like an ECS operator or EMR operator); but the documentation is aimed at python hackers running this on their laptop; all the defaults are aimed at this as well. This is a problem because essentially all of that is wrong for a proper cloud native type environment. We've looked at quite a few third party repos for terraform, kubernetes, cloudformation, etc that try to fix this. Ultimately we ended up spending non trivial amounts of time on devops. Basically, this involved lots of problem solving for things that a combination of wrong, poorly documented, or misguided by default. Also, we're not done by a long shot.
2) The UX/UI is terrible and I don't use this word lightly. Think hudson/jenkins, 15 years ago (and technically that's unfair to good old Hudson because it never was this bad). It's a fair comparison because Jenkins kind of is a drop in replacement or at least a significant overlap in feature set. And it arguably has a better ecosystem for things like plugins. Absolutely everything in Airflow requires multiple clicks. Also you'll be doing CMD+R a lot as there is no concept of autorefresh. Lots of fiddly icons. And then there's this obsession with graphs and this being the most important thing ever. There are two separate graph views, only one of which has useful ways of getting to the logs (which never requires less than 4-5 mouse clicks). And of course the other view is the default under most links so you have to learn to click the tiny graph icon to get to the good stuff.
3) A lot of the defaults are wrong/misguided/annoying. Like catch up defaulting to true. There's this weird notion of tasks (dags in airflow speak) running on a cron pattern and requiring a start date in the past. Using a dynamic date is not recommended (i.e. now would be a sane default). So typically you just pick whatever fixed time in the past. When you turn a dag on it tries to 'backfill' from that date unless you set catchup to false. I don't know in what universe that's a sane default. Sure, I want to run this task 1000 times just because I unpaused it (everything is paused by default). There is no way to unschedule that. Did I mention the default parallism is 32. That in combination with the docker operator is a great way to instantly run out of memory (yep that happened to us).
4) The UI lacks ways to group tasks like by tag or folders, etc. This gets annoying quickly.
5) Dag configs as code in a weakly typed language without a good test harness leads to obvious problems. We've sort of gobbled together our own tests to somewhat mitigate repeated deploy screw ups.
6) implementing a worker architecture in a language that is still burdened with the global interpreter lock and that has no good support for either threading or light weight threads (aka co-routines) or doing things asynchronously, leads to a lot of complexity. The celery worker is a PITA to debug.
7) IMHO the python operator is a bad idea because it gives data scientists the wrong idea about, oh just install this library on every airflow host please so I can run my thingy. We use the Docker operator a lot and are switching to the ECS operator as soon as we can figure out how to run airflow in ECS (we currently have a snow flaky AMI running on ec2).
8) the logging UI is terrible compared to what I would normally use for logging. Looking at logs of task runs is kind of the core business the UI has to do.
9) Airflow has a DB where it keeps track of state. Any change to dags basically means this state gets stale pretty quickly. There's no sane way to get rid of this stale data other than a lot of command-line fiddling or just running some sql scripts directly against this db. I've manually deleted hundreds of jobs in the last month. Also there's no notion of having a sane default for number of execution runs to preserve. Likewise the there is built in way to clean up logs. Again, Jenkins/Hudson had that always. I have jobs that run every 10 minutes and absolutely no need to keep months of history on that.
There are more things I could list. Also, there are quite a few competing products; this is a very crowded space. I've given serious thought to using Spring Batch or even just firing up a Jenkins. Frankly the only reason we chose airflow is that it's easier for data scientists who are mostly only comfortable with python. So far, I've been disappointed with how complex and flaky this setup is.
If you go down the path of using it, think hard about which operators you are going to use and why. IMHO dockerizing tasks means that most of what Airflow does is just ensuring your dockerized tasks run. Limiting what it does is a good thing. Just because you can doesn't mean you should in airflow. IMHO most of the operators naturally lead to your airflow installs being snow flakes.
Not dockerizing means you are mixing code and orchestration. Just like installing dependencies on CI servers is not a great idea is also the reason why doing the same on an airflow system is a bad idea.
- opportune 6y agoOther than 5+6 this seems like basically a spec for a managed airflow product. So basically run Airflow on public cloud, manage all the operational bits, create a better UI, and fix some upstream bugs.
- rotten 6y agoWhich is something both Google Cloud Composer and Astronomer.io are trying to do.
- rywalker 6y ago1) An official Helm chart is coming very soon. We (Astronomer) have a commercial platform that aims to solve this completely including observability, configuration and deployment. Happy to team up to improve Airflow's open-source k8s story if you have some ideas. 2) Yes the UI is outdated, and not responsive. We're going to kick off a process to build a new modern UI in Q3 (a full-featured Swagger API is being built now, which the new UI will rely upon.) 3) I personally think catchup true is a fine default, but whatever. Generally when I launch a new DAG I want to generate some historical data using the DAG. 4) Airflow has tags now since 1.10.8 https://airflow.readthedocs.io/en/latest/howto/add-dag-tags.html https://airflow.readthedocs.io/en/latest/howto/add-dag-tags.... - we decided not to do folders. 5) That's true, but it also provides a low bar to entry. There are some guides written on unit testing DAGs, but I agree we should be a test-first community. On my roadmap. 6) Celery w/ KEDA is pretty nice - check out https://www.astronomer.io/blog/the-keda-autoscaler/ https://www.astronomer.io/blog/the-keda-autoscaler/ 7) I personally love the PythonOperator for simple DAGs. Agree that DockerOperator (KubernetesPodOperator if you're running Airflow in K8s) 8) Yes, logging UI will be improved in the UI rewrite. What's your favorite UI for this? 9) That feature is in the queue https://github.com/apache/airflow/issues/7911 https://github.com/apache/airflow/issues/7911
- blakeburch 6y agoI couldn't agree more with you on most of these points. You may be interested in trying out Shipyard (www.shipyardapp.com). Fair disclosure, I'm the co-founder. While we don't address all of these issues, we're building with these key focuses. - Simplicity is key. Data Teams should focus on creating solutions, not fighting infrastructure and limitations. - Workflows shouldn't change how code is written. Your code should run the same locally as on our platform, with no extra packages or proprietary setup files required. - Templates are a first-class object. The modern data pipeline should be built with repeatability in mind. - Data solutions should be usable and visible beyond the walls of technical teams. We're in a private beta and rapidly trying to improve the product. I would love to chat more if you're interested. Details in profile. For your specific problems: 1) We're cloud-native and handle hosting/scaling on our side. You don't have to worry about setup. Just log in and launch your code. 2) Our UI is pretty slick (built in AntD) and built to reduce the overwhelming options when setting jobs up. 3) If you want to run a job on-demand, just press "Run now". If you schedule a job, then change the schedule or status, we'll automatically add/update/remove the schedules. Other technical defaults aren't options right now because we're trying to abstract those choices away so Data Teams can just focus on building solutions that work. 4) We let you group your scripts into "Projects" (essentially folders) with a high level overview of the quantity of jobs, as well as how many recently failed or succeeded. 5) Workflows get made directly in the UI. This makes it easier for any less technical users to set up jobs on their own. We still have a lot of improvement to go in this area though. 6) Our product is written in Go (the language of the cloud). We don't force the worker to be in a language and managed by a process in that language. We manage at the process level. 7) Every job creates a new container on the fly, installing package dependencies that the user specifies. You can connect your scripts together without worrying about conflicting packages, conflicting language versions, or without needing to know how to make and manage Docker containers. 8) Not sure how we compare on the logging front. However, we separate out logs for each time a script runs so you're not having to search for a needle in a haystack. You can filter and search for specific logs in the UI.