3 ms·
That's definitely a good question, and one that I don't think is clearly addressed in the docs (so I might take the opportunity to turn this into a little blog
by tedmiston 9y ago
That's definitely a good question, and one that I don't think is clearly addressed in the docs (so I might take the opportunity to turn this into a little blog post later). tl;dr - Airflow can support running multiple Python versions.
Here's a three-pronged answer:
1. PythonOperator - Assuming you're running Airflow on 3.6, you can run your 3.6 functions with the normal PythonOperator.
2. PythonVirtualenvOperator - Then, if upgrading your 2.7 code to 3.6 is an option, I would; if that's a hurdle... You can run multiple Python versions mixed using the PythonVirtualenvOperator [1] to run the other tasks in dedicated 2.7 etc virtual environments. That's pretty simple if it satisfies your requirements, but it's not a foolproof solution if your reason for being on 2.7 is complex or some system-level dependency that requires more.
3. DockerOperator - If the setup requires more complex things to be installed, you can use the DockerOperator [2] and build out a simple Python Docker image e.g., [3]. This is the method we use to run other languages like Scala and JavaScript via Airflow.
Feel free to reach out via email if you'd like to discuss more or have questions - taylor [at] astronomer.io. We like to help people get started with our Airflow and have written a bit of content for it.
[1]: https://github.com/apache/incubator-airflow/blob/ba126926d7d8a3d8d5bae548c12b5ce4b2a9fbd8/airflow/operators/python_operator.py#L162 https://github.com/apache/incubator-airflow/blob/ba126926d7d...
[2]: https://airflow.apache.org/code.html?highlight=dockeroperator#airflow.operators.docker_operator.DockerOperator https://airflow.apache.org/code.html?highlight=dockeroperato...
[3]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/
- liveoneggs 9y agoI will check out the docker operator. I didn't want to drop down into the bash executor to run docker containers because it felt a little dirty. My dev team is excited to use airflow but, in my estimation, keeping them all in lockstep with a single requirements.txt seems a little far fetched. ;)
- tedmiston 9y agoIt definitely feels weird at first but it's also worked well for us running millions of dockerized tasks. :) Yeah, I can see where you're coming from there. Some people will run multiple separate Airflow instances as another alternative solution.