5 ms·

Yeah, no, your impression is correct. Apache is and always has been a bit of an open source graveyard. There's counter-examples but even they have me worried.

Yeah, no, your impression is correct. Apache is and always has been a bit of an open source graveyard. There's counter-examples but even they have me worried. Airflow, originating from AirBNB, is now Apache Airflow: https://github.com/apache/incubator-airflow https://github.com/apache/incubator-airflow -- And I've been super hesitant to throw weight at it because of that, even though it looks actively maintained.

liveoneggs 9y ago
airflow is too tightly integrated into a single project/requirements.txt for my tastes. It should do less.
- tedmiston 9y ago
  Can you explain your position a bit more, like which parts you think would be better decoupled?
  - liveoneggs 9y ago
    I like airflow the dag/re-run/cron stuff but have a project running in py2.7 and one in 3.6, what do I do?
    - tedmiston 9y ago
      That's definitely a good question, and one that I don't think is clearly addressed in the docs (so I might take the opportunity to turn this into a little blog post later). tl;dr - Airflow can support running multiple Python versions. Here's a three-pronged answer: 1. PythonOperator - Assuming you're running Airflow on 3.6, you can run your 3.6 functions with the normal PythonOperator. 2. PythonVirtualenvOperator - Then, if upgrading your 2.7 code to 3.6 is an option, I would; if that's a hurdle... You can run multiple Python versions mixed using the PythonVirtualenvOperator [1] to run the other tasks in dedicated 2.7 etc virtual environments. That's pretty simple if it satisfies your requirements, but it's not a foolproof solution if your reason for being on 2.7 is complex or some system-level dependency that requires more. 3. DockerOperator - If the setup requires more complex things to be installed, you can use the DockerOperator [2] and build out a simple Python Docker image e.g., [3]. This is the method we use to run other languages like Scala and JavaScript via Airflow. Feel free to reach out via email if you'd like to discuss more or have questions - taylor [at] astronomer.io. We like to help people get started with our Airflow and have written a bit of content for it. [1]: https://github.com/apache/incubator-airflow/blob/ba126926d7d8a3d8d5bae548c12b5ce4b2a9fbd8/airflow/operators/python_operator.py#L162 https://github.com/apache/incubator-airflow/blob/ba126926d7d... [2]: https://airflow.apache.org/code.html?highlight=dockeroperator#airflow.operators.docker_operator.DockerOperator https://airflow.apache.org/code.html?highlight=dockeroperato... [3]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/
      - liveoneggs 9y ago
        I will check out the docker operator. I didn't want to drop down into the bash executor to run docker containers because it felt a little dirty. My dev team is excited to use airflow but, in my estimation, keeping them all in lockstep with a single requirements.txt seems a little far fetched. ;)
        tedmiston 9y ago
        It definitely feels weird at first but it's also worked well for us running millions of dockerized tasks. :) Yeah, I can see where you're coming from there. Some people will run multiple separate Airflow instances as another alternative solution.
tedmiston 9y ago
Airflow is extremely active — check the mailing list, Gitter chat, or even GitHub contributions. My company, Astronomer, also provides a hosted version of Airflow [1] if SaaS is more you're thing. [1]: https://www.astronomer.io/airflow https://www.astronomer.io/airflow
- tedmiston 9y ago
  %s/you're/your /me shudders
fphhotchips 9y ago
That's partially because of the ecosystem it plays in. Plenty of people who won't touch a Hadoop/Big Data ecosystem tool unless it has 'Apache' in front of it.
rjurney 9y ago
What you are seeing as "dying" is actually "widespread adoption and maturity", only you've moved on because you're "latest and greatest." Nothing wrong with that, but doesn't mean anything has died. Only your interest has.
- pbhjpbhj 9y ago
  So Wave, say, is now a mature and widely adopted product?
  - rjurney 9y ago
    It is not a global statement :)

liveoneggs 9y ago

airflow is too tightly integrated into a single project/requirements.txt for my tastes. It should do less.

tedmiston 9y ago
Can you explain your position a bit more, like which parts you think would be better decoupled?
- liveoneggs 9y ago
  I like airflow the dag/re-run/cron stuff but have a project running in py2.7 and one in 3.6, what do I do?
  - tedmiston 9y ago
    That's definitely a good question, and one that I don't think is clearly addressed in the docs (so I might take the opportunity to turn this into a little blog post later). tl;dr - Airflow can support running multiple Python versions. Here's a three-pronged answer: 1. PythonOperator - Assuming you're running Airflow on 3.6, you can run your 3.6 functions with the normal PythonOperator. 2. PythonVirtualenvOperator - Then, if upgrading your 2.7 code to 3.6 is an option, I would; if that's a hurdle... You can run multiple Python versions mixed using the PythonVirtualenvOperator [1] to run the other tasks in dedicated 2.7 etc virtual environments. That's pretty simple if it satisfies your requirements, but it's not a foolproof solution if your reason for being on 2.7 is complex or some system-level dependency that requires more. 3. DockerOperator - If the setup requires more complex things to be installed, you can use the DockerOperator [2] and build out a simple Python Docker image e.g., [3]. This is the method we use to run other languages like Scala and JavaScript via Airflow. Feel free to reach out via email if you'd like to discuss more or have questions - taylor [at] astronomer.io. We like to help people get started with our Airflow and have written a bit of content for it. [1]: https://github.com/apache/incubator-airflow/blob/ba126926d7d8a3d8d5bae548c12b5ce4b2a9fbd8/airflow/operators/python_operator.py#L162 https://github.com/apache/incubator-airflow/blob/ba126926d7d... [2]: https://airflow.apache.org/code.html?highlight=dockeroperator#airflow.operators.docker_operator.DockerOperator https://airflow.apache.org/code.html?highlight=dockeroperato... [3]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/
    - liveoneggs 9y ago
      I will check out the docker operator. I didn't want to drop down into the bash executor to run docker containers because it felt a little dirty. My dev team is excited to use airflow but, in my estimation, keeping them all in lockstep with a single requirements.txt seems a little far fetched. ;)
      - tedmiston 9y ago
        It definitely feels weird at first but it's also worked well for us running millions of dockerized tasks. :) Yeah, I can see where you're coming from there. Some people will run multiple separate Airflow instances as another alternative solution.

tedmiston 9y ago

Airflow is extremely active — check the mailing list, Gitter chat, or even GitHub contributions. My company, Astronomer, also provides a hosted version of Airflow [1] if SaaS is more you're thing. [1]: https://www.astronomer.io/airflow https://www.astronomer.io/airflow

tedmiston 9y ago
%s/you're/your /me shudders

fphhotchips 9y ago

That's partially because of the ecosystem it plays in. Plenty of people who won't touch a Hadoop/Big Data ecosystem tool unless it has 'Apache' in front of it.

rjurney 9y ago

What you are seeing as "dying" is actually "widespread adoption and maturity", only you've moved on because you're "latest and greatest." Nothing wrong with that, but doesn't mean anything has died. Only your interest has.

pbhjpbhj 9y ago
So Wave, say, is now a mature and widely adopted product?
- rjurney 9y ago
  It is not a global statement :)