3 ms·

I'm actually considering using Airflow. Have never used it before, and I have the impression that setting up the required infrastructure could be problematic.

I'm actually considering using Airflow. Have never used it before, and I have the impression that setting up the required infrastructure could be problematic. Since a lot of you use Airflow, I am curious about your experience with it: 1. Are you hosting Airflow yourselves or using a managed service? 1. a. If managed, which one? (Google Cloud Composer, Astronomer.io, something else?) 1. b. If self-hosted, how difficult was the setup? It seems daunting to get a stable setup (external database, rabbit or redis, etc.). 2. Do you use one operator (DockerOperator looks like the right choice) or do you allow yourself freedom in operators? Do you build your own? 3. How do you pass data from one task to the next? Do the tasks themselves have to be aware of external storage conventions or do you use the built-in xcom mechanism? It seems like xcom stores messages in the database, so you run the risk of blowing through storage capacity this way?

pyrophane 6y ago
1. Managed, Cloud Composer. Cloud Composer is getting there. It feels much less buggy then just 8 months ago when I started using it, and it is improving rather quickly. One downside with Composer, though, is that it must be run in its own GKE cluster, and it deploys the Airflow UI to App Engine. These two things can make it a bit of a pain to use alongside infrastructure deployed into another GKE cluster if you need the two to interact. I would probably still recommend Composer over deploying your own Airflow into GKE, as having it managed is nice. 2. Freedom. For some tasks we run containers in GKE, for other we use things like the PythonOperator or PostgresOperator. A note here: Using containers with Airflow is not trivial. In addition to needing some CI process to manage image building/deployment, having the ability to develop and test DAGs locally takes some extra work. I would only recommend it if you are already invested in containers and are willing to devote the time to ops to get it all working. 3. X-com is useful for small amounts of data, like if one task needs to pass a file path, IDs, or other parameters to a downstream task. For everything else have a task write its output to something like S3 or a database that another task will read from. All in all, I would say use Airflow if you need the visibility and dependency management. Don't use it if you could get away with something like cron and some scripts or a simple pool of celery workers. Also, don't use it if your workflows are highly dynamic. For example, if you have a situation where you need to run a task to get a list of things, then span x downstream tasks based on the contents of the list. Airflow wants the shape of the DAG to be defined before it is run. Hope that helps.
- zomglings 6y ago
  Thank you, it helps a lot!
- hn2017 6y ago
  Your last point about highly dynamic workflows was a particular pain point for me and I think for many others. One recommendation for Airflow is to create a list of use cases with sample DAGs to show best practices.
mrshu 6y ago
1. Managed, on AWS ECS -- it was not that difficult to set up using AWS services (Aurora for the DB, ElastiCache for Redis) 2. Freedom. We generally use PythonOperators but it is not uncommon to run containers as well. I agree with pyrophane, setting up containerized operators really is a non-trivial and far from straightforward to test locally. Still, it seems worth doing, particularly if you do not want various execution to affect one another. 3. Again, echoing what pyrophane said, a custom solution is needed for anything that's more than a couple hundred bytes in size. There even exists (now mostly abandoned) plugin that allows you to streamline the whole process: https://github.com/industrydive/fileflow https://github.com/industrydive/fileflow Writing directly to something like S3 is almost always sufficient in combination with passing the path for the file from one task to another. Having said that, I would encourage you to try it out, even if the setup may sound daunting at first. If you can model your tasks as DAGs whose graphs are known in advance, I would argue it almost always makes sense because you get many things "for free" that were not mentioned before: logging, backfill, standard handling of connections/secrets and a ton of metrics in a nice UI that helps a lot with the visibility part.
ies7 6y ago
1. self hosted on ec2. 1b. Pip install airflow[all|what you need] The airflow itself is easy to install. I’d say that installing the external tools is also easy. I believe installing pg, redis, or celery should be categorized in easy. It’s not the kafka or k8s level of installation. 2. Freedom. 3. Custom scripts

pyrophane 6y ago

1. Managed, Cloud Composer. Cloud Composer is getting there. It feels much less buggy then just 8 months ago when I started using it, and it is improving rather quickly. One downside with Composer, though, is that it must be run in its own GKE cluster, and it deploys the Airflow UI to App Engine. These two things can make it a bit of a pain to use alongside infrastructure deployed into another GKE cluster if you need the two to interact. I would probably still recommend Composer over deploying your own Airflow into GKE, as having it managed is nice. 2. Freedom. For some tasks we run containers in GKE, for other we use things like the PythonOperator or PostgresOperator. A note here: Using containers with Airflow is not trivial. In addition to needing some CI process to manage image building/deployment, having the ability to develop and test DAGs locally takes some extra work. I would only recommend it if you are already invested in containers and are willing to devote the time to ops to get it all working. 3. X-com is useful for small amounts of data, like if one task needs to pass a file path, IDs, or other parameters to a downstream task. For everything else have a task write its output to something like S3 or a database that another task will read from. All in all, I would say use Airflow if you need the visibility and dependency management. Don't use it if you could get away with something like cron and some scripts or a simple pool of celery workers. Also, don't use it if your workflows are highly dynamic. For example, if you have a situation where you need to run a task to get a list of things, then span x downstream tasks based on the contents of the list. Airflow wants the shape of the DAG to be defined before it is run. Hope that helps.

zomglings 6y ago
Thank you, it helps a lot!
hn2017 6y ago
Your last point about highly dynamic workflows was a particular pain point for me and I think for many others. One recommendation for Airflow is to create a list of use cases with sample DAGs to show best practices.

mrshu 6y ago

1. Managed, on AWS ECS -- it was not that difficult to set up using AWS services (Aurora for the DB, ElastiCache for Redis) 2. Freedom. We generally use PythonOperators but it is not uncommon to run containers as well. I agree with pyrophane, setting up containerized operators really is a non-trivial and far from straightforward to test locally. Still, it seems worth doing, particularly if you do not want various execution to affect one another. 3. Again, echoing what pyrophane said, a custom solution is needed for anything that's more than a couple hundred bytes in size. There even exists (now mostly abandoned) plugin that allows you to streamline the whole process: https://github.com/industrydive/fileflow https://github.com/industrydive/fileflow Writing directly to something like S3 is almost always sufficient in combination with passing the path for the file from one task to another. Having said that, I would encourage you to try it out, even if the setup may sound daunting at first. If you can model your tasks as DAGs whose graphs are known in advance, I would argue it almost always makes sense because you get many things "for free" that were not mentioned before: logging, backfill, standard handling of connections/secrets and a ton of metrics in a nice UI that helps a lot with the visibility part.

ies7 6y ago

1. self hosted on ec2. 1b. Pip install airflow[all|what you need] The airflow itself is easy to install. I’d say that installing the external tools is also easy. I believe installing pg, redis, or celery should be categorized in easy. It’s not the kafka or k8s level of installation. 2. Freedom. 3. Custom scripts