6 ms·

While the Apache Foundation hosts a lot of great projects, it also looks to me like a place where big projects go to die. I'm thinking of the whole Apache Open

by tux3 9y ago

While the Apache Foundation hosts a lot of great projects, it also looks to me like a place where big projects go to die. I'm thinking of the whole Apache Open-Office fiasco, among other things. Looking at their list of projects sorted by number of committers[0], event at the top I don't get the idea that most of those have a lot of momentum. Do I just have a completely wrong impression? I'm curious if there's something to learn from this. [0]: https://projects.apache.org/projects.html?number https://projects.apache.org/projects.html?number

scrollaway 9y ago
Yeah, no, your impression is correct. Apache is and always has been a bit of an open source graveyard. There's counter-examples but even they have me worried. Airflow, originating from AirBNB, is now Apache Airflow: https://github.com/apache/incubator-airflow https://github.com/apache/incubator-airflow -- And I've been super hesitant to throw weight at it because of that, even though it looks actively maintained.
- liveoneggs 9y ago
  airflow is too tightly integrated into a single project/requirements.txt for my tastes. It should do less.
  - tedmiston 9y ago
    Can you explain your position a bit more, like which parts you think would be better decoupled?
    - liveoneggs 9y ago
      I like airflow the dag/re-run/cron stuff but have a project running in py2.7 and one in 3.6, what do I do?
      - tedmiston 9y ago
        That's definitely a good question, and one that I don't think is clearly addressed in the docs (so I might take the opportunity to turn this into a little blog post later). tl;dr - Airflow can support running multiple Python versions. Here's a three-pronged answer: 1. PythonOperator - Assuming you're running Airflow on 3.6, you can run your 3.6 functions with the normal PythonOperator. 2. PythonVirtualenvOperator - Then, if upgrading your 2.7 code to 3.6 is an option, I would; if that's a hurdle... You can run multiple Python versions mixed using the PythonVirtualenvOperator [1] to run the other tasks in dedicated 2.7 etc virtual environments. That's pretty simple if it satisfies your requirements, but it's not a foolproof solution if your reason for being on 2.7 is complex or some system-level dependency that requires more. 3. DockerOperator - If the setup requires more complex things to be installed, you can use the DockerOperator [2] and build out a simple Python Docker image e.g., [3]. This is the method we use to run other languages like Scala and JavaScript via Airflow. Feel free to reach out via email if you'd like to discuss more or have questions - taylor [at] astronomer.io. We like to help people get started with our Airflow and have written a bit of content for it. [1]: https://github.com/apache/incubator-airflow/blob/ba126926d7d8a3d8d5bae548c12b5ce4b2a9fbd8/airflow/operators/python_operator.py#L162 https://github.com/apache/incubator-airflow/blob/ba126926d7d... [2]: https://airflow.apache.org/code.html?highlight=dockeroperator#airflow.operators.docker_operator.DockerOperator https://airflow.apache.org/code.html?highlight=dockeroperato... [3]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/
        liveoneggs 9y ago
        I will check out the docker operator. I didn't want to drop down into the bash executor to run docker containers because it felt a little dirty. My dev team is excited to use airflow but, in my estimation, keeping them all in lockstep with a single requirements.txt seems a little far fetched. ;)
        tedmiston 9y ago
        It definitely feels weird at first but it's also worked well for us running millions of dockerized tasks. :) Yeah, I can see where you're coming from there. Some people will run multiple separate Airflow instances as another alternative solution.
- tedmiston 9y ago
  Airflow is extremely active — check the mailing list, Gitter chat, or even GitHub contributions. My company, Astronomer, also provides a hosted version of Airflow [1] if SaaS is more you're thing. [1]: https://www.astronomer.io/airflow https://www.astronomer.io/airflow
  - tedmiston 9y ago
    %s/you're/your /me shudders
- fphhotchips 9y ago
  That's partially because of the ecosystem it plays in. Plenty of people who won't touch a Hadoop/Big Data ecosystem tool unless it has 'Apache' in front of it.
- rjurney 9y ago
  What you are seeing as "dying" is actually "widespread adoption and maturity", only you've moved on because you're "latest and greatest." Nothing wrong with that, but doesn't mean anything has died. Only your interest has.
  - pbhjpbhj 9y ago
    So Wave, say, is now a mature and widely adopted product?
    - rjurney 9y ago
      It is not a global statement :)
squeeeeeeeeeee 9y ago
Next in line: Netbeans.
- chapill 9y ago
  I've recently discovered Netbeans and I like it. It's currently the only Java IDE I can get running on aarch64. But I know what you're saying. The contributors that are left sound rather desperate for help.
chapill 9y ago
That has a lot to do with the license. It should be more copyleft. Permissive licenses attract leeches who do nothing but complain to overworked contributors. "Over the years, I've become convinced that the BSD license is great for code you don't care about" -- Linus Torvalds https://www.cio.com/article/3112582/linux/linus-torvalds-says-gpl-was-defining-factor-in-linuxs-success.html https://www.cio.com/article/3112582/linux/linus-torvalds-say...
jaredsohn 9y ago
There is a meme associated with this; search for 'apache projects go to die'
liveoneggs 9y ago
Where are: httpd, lucene, and solr in that list?
Cyberdog 9y ago
Did Wave ever really live, though? I know it had its vocal fans for whatever reason, but most people never understood what it was, even if they had heard of it and tried to use it (like myself).
- randomerr 9y ago
  Wave was a layer that live on top of a modified Jabber/XMPP chat server. It sent extra meta data to your messaging client. Depending on your client and meta data sent your client could download JavaScipt applets you authorized. The applet would render the meta data into something like an iFrame to show a Google Document or a PacMan game. Also you could create a separate messaging thread from a single comment but still have a link to parent thread (ala #Slack now.) The branching comments, embedded documents, and ability to create mini applets were the big selling points. But the feature were never well explained and internal implementation was a nightmare. As people could still hook in Jabber with client it was also creating issues with spamming and hacking.
- dreamfactored 9y ago
  The interesting bit of Wave wasn't the product but the underlying framework components they created for it e.g. salmon protocol and the pubsub system. They made many of the components needed to create an open source federated social networking system where separate implementations could interact with each other. So for example your user on your company's twitter/fb hybrid could have threaded conversations with other users across other organisation's fb clones (obviously with access control). This could have been exactly the killer functionality to topple centralised proprietary Twitter and Facebook. Google should have thrown everything into it, could have turned out as impactful as hypertext.
jburgess777 9y ago
That projects site seems to give a particularly pessimistic view. I looked at the 'traffic server' project[1] (a web cache) and it says the last release was in 2012, but the real project website[2] shows there has been a steady stream of releases, the last one just a few weeks ago. 1: https://projects.apache.org/project.html?trafficserver-traffic_server https://projects.apache.org/project.html?trafficserver-traff... 2: http://trafficserver.apache.org/downloads http://trafficserver.apache.org/downloads
rectang 9y ago
> Do I just have a completely wrong impression? You have a completely wrong impression. Projects in the wild die all the time, and the process is not necessarily neat and tidy. At the ASF, projects must be actively retired, and there are a sequence of actions which must be taken to move the project into the "Attic". Retirement gives all stakeholders a defined close of business. All software projects have a life cycle. What is notable about ASF projects is that the end of the life cycle is comparatively orderly. I wish the Wave folks well as they move on to whatever's next for them.
- amyjess 9y ago
  And this, of course, leads to selection bias. You hear more about the ASF retiring projects because the ASF has a whole process around it that's publicized. You don't hear too much about some one-man project that stops updating its GitHub all of a sudden. You hear about ASF projects dying becuase the ASF wants people to hear about it.
- pbhjpbhj 9y ago
  Everything you said there supports the parents notion that ASF is where big projects go to die. You just called it retirement.

by tux3 9y ago

scrollaway 9y ago

Yeah, no, your impression is correct. Apache is and always has been a bit of an open source graveyard. There's counter-examples but even they have me worried. Airflow, originating from AirBNB, is now Apache Airflow: https://github.com/apache/incubator-airflow https://github.com/apache/incubator-airflow -- And I've been super hesitant to throw weight at it because of that, even though it looks actively maintained.

liveoneggs 9y ago
airflow is too tightly integrated into a single project/requirements.txt for my tastes. It should do less.
- tedmiston 9y ago
  Can you explain your position a bit more, like which parts you think would be better decoupled?
  - liveoneggs 9y ago
    I like airflow the dag/re-run/cron stuff but have a project running in py2.7 and one in 3.6, what do I do?
    - tedmiston 9y ago
      That's definitely a good question, and one that I don't think is clearly addressed in the docs (so I might take the opportunity to turn this into a little blog post later). tl;dr - Airflow can support running multiple Python versions. Here's a three-pronged answer: 1. PythonOperator - Assuming you're running Airflow on 3.6, you can run your 3.6 functions with the normal PythonOperator. 2. PythonVirtualenvOperator - Then, if upgrading your 2.7 code to 3.6 is an option, I would; if that's a hurdle... You can run multiple Python versions mixed using the PythonVirtualenvOperator [1] to run the other tasks in dedicated 2.7 etc virtual environments. That's pretty simple if it satisfies your requirements, but it's not a foolproof solution if your reason for being on 2.7 is complex or some system-level dependency that requires more. 3. DockerOperator - If the setup requires more complex things to be installed, you can use the DockerOperator [2] and build out a simple Python Docker image e.g., [3]. This is the method we use to run other languages like Scala and JavaScript via Airflow. Feel free to reach out via email if you'd like to discuss more or have questions - taylor [at] astronomer.io. We like to help people get started with our Airflow and have written a bit of content for it. [1]: https://github.com/apache/incubator-airflow/blob/ba126926d7d8a3d8d5bae548c12b5ce4b2a9fbd8/airflow/operators/python_operator.py#L162 https://github.com/apache/incubator-airflow/blob/ba126926d7d... [2]: https://airflow.apache.org/code.html?highlight=dockeroperator#airflow.operators.docker_operator.DockerOperator https://airflow.apache.org/code.html?highlight=dockeroperato... [3]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/
      - liveoneggs 9y ago
        I will check out the docker operator. I didn't want to drop down into the bash executor to run docker containers because it felt a little dirty. My dev team is excited to use airflow but, in my estimation, keeping them all in lockstep with a single requirements.txt seems a little far fetched. ;)
        tedmiston 9y ago
        It definitely feels weird at first but it's also worked well for us running millions of dockerized tasks. :) Yeah, I can see where you're coming from there. Some people will run multiple separate Airflow instances as another alternative solution.
tedmiston 9y ago
Airflow is extremely active — check the mailing list, Gitter chat, or even GitHub contributions. My company, Astronomer, also provides a hosted version of Airflow [1] if SaaS is more you're thing. [1]: https://www.astronomer.io/airflow https://www.astronomer.io/airflow
- tedmiston 9y ago
  %s/you're/your /me shudders
fphhotchips 9y ago
That's partially because of the ecosystem it plays in. Plenty of people who won't touch a Hadoop/Big Data ecosystem tool unless it has 'Apache' in front of it.
rjurney 9y ago
What you are seeing as "dying" is actually "widespread adoption and maturity", only you've moved on because you're "latest and greatest." Nothing wrong with that, but doesn't mean anything has died. Only your interest has.
- pbhjpbhj 9y ago
  So Wave, say, is now a mature and widely adopted product?
  - rjurney 9y ago
    It is not a global statement :)

squeeeeeeeeeee 9y ago

Next in line: Netbeans.

chapill 9y ago
I've recently discovered Netbeans and I like it. It's currently the only Java IDE I can get running on aarch64. But I know what you're saying. The contributors that are left sound rather desperate for help.

chapill 9y ago

That has a lot to do with the license. It should be more copyleft. Permissive licenses attract leeches who do nothing but complain to overworked contributors. "Over the years, I've become convinced that the BSD license is great for code you don't care about" -- Linus Torvalds https://www.cio.com/article/3112582/linux/linus-torvalds-says-gpl-was-defining-factor-in-linuxs-success.html https://www.cio.com/article/3112582/linux/linus-torvalds-say...

jaredsohn 9y ago

There is a meme associated with this; search for 'apache projects go to die'

liveoneggs 9y ago

Where are: httpd, lucene, and solr in that list?

Cyberdog 9y ago

Did Wave ever really live, though? I know it had its vocal fans for whatever reason, but most people never understood what it was, even if they had heard of it and tried to use it (like myself).

randomerr 9y ago
Wave was a layer that live on top of a modified Jabber/XMPP chat server. It sent extra meta data to your messaging client. Depending on your client and meta data sent your client could download JavaScipt applets you authorized. The applet would render the meta data into something like an iFrame to show a Google Document or a PacMan game. Also you could create a separate messaging thread from a single comment but still have a link to parent thread (ala #Slack now.) The branching comments, embedded documents, and ability to create mini applets were the big selling points. But the feature were never well explained and internal implementation was a nightmare. As people could still hook in Jabber with client it was also creating issues with spamming and hacking.
dreamfactored 9y ago
The interesting bit of Wave wasn't the product but the underlying framework components they created for it e.g. salmon protocol and the pubsub system. They made many of the components needed to create an open source federated social networking system where separate implementations could interact with each other. So for example your user on your company's twitter/fb hybrid could have threaded conversations with other users across other organisation's fb clones (obviously with access control). This could have been exactly the killer functionality to topple centralised proprietary Twitter and Facebook. Google should have thrown everything into it, could have turned out as impactful as hypertext.

jburgess777 9y ago

That projects site seems to give a particularly pessimistic view. I looked at the 'traffic server' project[1] (a web cache) and it says the last release was in 2012, but the real project website[2] shows there has been a steady stream of releases, the last one just a few weeks ago. 1: https://projects.apache.org/project.html?trafficserver-traffic_server https://projects.apache.org/project.html?trafficserver-traff... 2: http://trafficserver.apache.org/downloads http://trafficserver.apache.org/downloads

rectang 9y ago

> Do I just have a completely wrong impression? You have a completely wrong impression. Projects in the wild die all the time, and the process is not necessarily neat and tidy. At the ASF, projects must be actively retired, and there are a sequence of actions which must be taken to move the project into the "Attic". Retirement gives all stakeholders a defined close of business. All software projects have a life cycle. What is notable about ASF projects is that the end of the life cycle is comparatively orderly. I wish the Wave folks well as they move on to whatever's next for them.

amyjess 9y ago
And this, of course, leads to selection bias. You hear more about the ASF retiring projects because the ASF has a whole process around it that's publicized. You don't hear too much about some one-man project that stops updating its GitHub all of a sudden. You hear about ASF projects dying becuase the ASF wants people to hear about it.
pbhjpbhj 9y ago
Everything you said there supports the parents notion that ASF is where big projects go to die. You just called it retirement.