4 ms·

I setup airflow for production at a consulting gig years ago. Airflow is an abstraction and a good abstraction. I don’t understand why these blog posts try to

by ibash 2y ago

I setup airflow for production at a consulting gig years ago. Airflow is an abstraction and a good abstraction. I don’t understand why these blog posts try to make it seem more complex and confusing than it actually is… Is it trying to justify the cost of a managed service? Or something else?

seeyam 2y ago
At scale it's probably to ensure your environment is capable of handling as many DAGs as possible. also as a data engineer, not knowing why a task isn't running can be frustrating. so this might help those understand and resolve
gregw2 2y ago
This article is blog post describing concurrency on Google's managed service version of Airflow, Compose, on Google's blog. While I don't have hands on with that, I have some hands on experience with the AWS managed service equivalent, Managed Workflows for Apache Airflow (MWAA) and its slightly similar documentation and/or challenges. We migrated 300+ jobs from a different but roughly similar orchestration tool to Airflow DAGs (and some jobs had as many as 50 parallel tasks for extracting data). If my experience with MWAA is any guide, there are enough minor differences between a managed-service Airflow vs self-managed Airflow that -- particularly for someone new to Airflow and concerned with scaling or performance issues for which SaaS platforms give you a narrower set of lower-effort knobs -- its probably worth having a tutorial to help people out so they don't have to mentally understand the full Airflow documentation and which bits of it do or don't, might or might not be able to be applied to them. (Managed Airflow does lock down some things which it has to for security/multi-tenancy reasons, and then has some differences to support "autoscaling" beyond what you get with self-hosted single-server Airflow. I didn't really see any aspects of the differences that I considered to be vendor attempts at vendor lockin, just needed differences for a SaaS product.) That sort of coherent-just-for-Google-Compose viewpoint seems to be what this blog post seems to focus on. I feel like we didn't get that coherent documentation from AWS's Airflow SaaS. It was mediocre (but passable) for a team new to Airflow. We had to learn when to read one set of documentation, when the other, and how to infer and map across the two to get a wholistic view of Airflow sufficient to solve our problems. It actually is nice under such situations if the cloud vendor provides coherent complete documentation even if it recapitulates the core project documentation or is somewhat redundant.

by ibash 2y ago

seeyam 2y ago

At scale it's probably to ensure your environment is capable of handling as many DAGs as possible. also as a data engineer, not knowing why a task isn't running can be frustrating. so this might help those understand and resolve

gregw2 2y ago

This article is blog post describing concurrency on Google's managed service version of Airflow, Compose, on Google's blog. While I don't have hands on with that, I have some hands on experience with the AWS managed service equivalent, Managed Workflows for Apache Airflow (MWAA) and its slightly similar documentation and/or challenges. We migrated 300+ jobs from a different but roughly similar orchestration tool to Airflow DAGs (and some jobs had as many as 50 parallel tasks for extracting data). If my experience with MWAA is any guide, there are enough minor differences between a managed-service Airflow vs self-managed Airflow that -- particularly for someone new to Airflow and concerned with scaling or performance issues for which SaaS platforms give you a narrower set of lower-effort knobs -- its probably worth having a tutorial to help people out so they don't have to mentally understand the full Airflow documentation and which bits of it do or don't, might or might not be able to be applied to them. (Managed Airflow does lock down some things which it has to for security/multi-tenancy reasons, and then has some differences to support "autoscaling" beyond what you get with self-hosted single-server Airflow. I didn't really see any aspects of the differences that I considered to be vendor attempts at vendor lockin, just needed differences for a SaaS product.) That sort of coherent-just-for-Google-Compose viewpoint seems to be what this blog post seems to focus on. I feel like we didn't get that coherent documentation from AWS's Airflow SaaS. It was mediocre (but passable) for a team new to Airflow. We had to learn when to read one set of documentation, when the other, and how to infer and map across the two to get a wholistic view of Airflow sufficient to solve our problems. It actually is nice under such situations if the cloud vendor provides coherent complete documentation even if it recapitulates the core project documentation or is somewhat redundant.