9 ms·

Amazon’s quiet open source revolution

126 points by unripe_syntax 4y ago

tyingq 4y ago
Directionally good, though I'm sure there's a decision tree that's mostly driven by how it benefits AWS. Something like "is committing this back going to save us time maintaining it separately without giving away IP we care about?".
- ghaff 4y ago
  It's probably not so much IP that they care about as IP that gives them some competitive advantage by keeping it secret. You can probably overstate the degree to which Amazon didn't contribute to and interact with open source communities 5+ years ago (and probably also overstate the degree to which there's been a complete sea change) but, as you say I think it's fair to say there's been a directional shift with more substantive activity in addition to what I'd mostly describe as talking the talk previously. The industry as a whole has also developed into this incredibly complicated coopetition landscape especially around containers in the past decade.
- nazgulsenpai 4y ago
  I would say that's true for any company that contributes to open source projects.
  - tyingq 4y ago
    Yes, though there's some things that are somewhat unique where Amazon is competing with open source products. For example, they have to offer MKS, but would rather you use Kinesis, EventBridge, etc. So perhaps less incentive to make Kafka better for everyone.
- jrochkind1 4y ago
  I feel like that kind of self-interest is the origin of open source, and always what has powered it when it's working. The original apache httpd, for instance, was written by people working at different organizations collaborating together to build the web server that they needed. It wasn't something people did as just a hobby, although the individual contributors may also have had some altruistic and certainly cooperative motivations. It definitely wasn't something people did motivated by trying to make money off of selling apache httpd in some way. Open source always requires self-interest to fuel it. The original model was people collaborating across organizations because it provided something useful to their organization -- exactly this. When this stops happening, because organizations cut budgets to the bone; because they gatekeep anything that might benefit competitors as well as themselves; because they can free-load off existing open source instead -- what other source of self-interest is there? One is people trying to profit off of the open source itself, with "open core" models, or models where the license can appear open but tries to ensure that the creators maintain a monopoly on any profitable aspects. Or, we can just get lack of investment in open source everyone is trying to free-load off of, withering open source with burnt out maintainers. Companies committing to open source because it saves them the time of maintaining it separately is the best case, and describes the environment of the "golden age" of open source.
  - pydry 4y ago
    I met an open source maintainer once working on a very popular tool everybody uses who was practically impoverished. From talking to him it became apparent that he was routinely being taken advantage of across all spheres of his life. It made me wonder how much valuable open source is powered, quietly, by people who are just used to being exploited.
    - pabs3 4y ago
      Probably almost all of it, except large projects with lots of paid devs, like Linux or RedHat.
  - taftster 4y ago
    This is insightful. > Open source always requires self-interest to fuel it. That self-interest doesn't necessarily have to be profit motivated either. As one of the replies to your comment suggest, probably the majority of open source is maintained as a hobby by an invested/interested user. Open source does not have to have a profit motive to make sense. But likewise, a profit motive is also congruent with open source. It greases a lot of axles, to put it nicely. There's definitely some disparity in open source contributors in terms of money being made, almost like the income gap between wealth and poverty in general. I wish we could do better at fixing that and helping the "little guy" that is contributing their soul into a project. Still though, open source is driven by "selfish" means, in whatever form that takes.
- pkkm 4y ago
  Sure, but why does that matter? A lot of good software got developed by itch-scratching.
Dowwie 4y ago
Their s2n projects seem like valuable contributions to the ecosystem. Strong, well maintained libraries supporting networking and cryptography are important.
phpisthebest 4y ago
I wish that same drive would come to their devices division which do the absolutely bear minimum not to be sued for GPL Violation when they use GPL code on their Fire Devices Edit: Also doesn't AWS have a history of Microsoft style EEE, where they embrace an open source project, make a service from it, extend that service in a non-open way and then push people to use their extended non-open service instead of improving upstream? Hasn't this resulted in a number of projects changing their licensing to prevent it?
- smatija 4y ago
  I think ElasticSearch license change was biggest such change, as was discussed previously on HN: https://news.ycombinator.com/item?id=25776657 https://news.ycombinator.com/item?id=25776657
  - pydry 4y ago
    Their "obsession" with contributing isnt really in evidence on their fork: https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da10b894b728b78e77e5e936a6427a7803b2865a0/68747470733a2f2f63646e2e747467746d656469612e636f6d2f726d732f6f6e6c696e65696d616765732f636c6f7564636f6d707574696e672d6769746875625f636f64655f636f6d6d6974732d662e706e67 https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da... Worth bearing in mind that the only reason fork even exists was so that they didnt have to pay Elastic a cut when selling hosted ElasticSearch. If they cared about open source they could start by not undermining the business models of companies that do more than just pretend to support it.
    - TravelPiglet 4y ago
      Having to pay for hosting an open source product? Isn’t Amazon doing what they should in this case where a company tries to block any competition.
      - pydry 4y ago
        No. Amazon was abusing their enormous market power to squeeze the company that made this software out of the market of hosting it. (Not to mention lying about "partnering" with them) If allowed to continue this would kill the project. If only Amazon can monetize it, it doesnt get developed. The open source project then dies. Amazon is no more doing the right thing here than they were when they destroyed diapers.com with a strategy of sustained predatory pricing. The fact that they want a pat on the back for being "pro open source" now after pulling this kind of crap is just icing on the cake.
andrewstuart 4y ago
One of the things that led me to lose interest in AWS was its attitude to open source.
pydry 4y ago
>Though it was always incorrect to lambast AWS for “strip-mining” open source It really wasn't. That was their whole approach to Elastic. The tantrum they threw at Elastic when they relicensed was particularly cringeworthy.
- prepend 4y ago
  They didn’t throw a tantrum. They just forked and continued. A perfectly reasonable approach. I think it’s important to describe things accurately. Disliking something isn’t a “tantrum.” I think amazons blog posts were pretty normal speech and tl:dr’d to “we don’t like it, so now we forked it and will maintain under Apache.” And they did that, so it’s actually kind of nice as I can’t use ElasticSearch under its new license but can use open search.
  - pydry 4y ago
    Tantrum is pretty accurate I think. They voiced their extreme displeasure in a war of words with Elastic. They knew that the relicensing was targeted 100% at their gravy train and they condescendingly declared that they would "step up on behalf of open source" while they stepped up exclusively to prevent Elastic getting a cut of their ElasticSearch hosting profits. The fork is really half hearted with about 1/10th the resources Elastic dedicates to the mainline branch: https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da... Meanwhile, the only effective licensing difference is that you can sell hosting with their fork - something that only benefits them.
    - belval 4y ago
      A lot of forks are created for similar reasons, it's not a fight between the great enlightened and the vile defilers. It's a for-profit (Elastic) wanting more control over their product (ElasticSearch) while also making ElasticSearch open-source under a permissive license that they used to kick-start their growth and another for-profit (Amazon) leveraging the open-source license to offer the product as a service to their customer base. When they had a disagreement Amazon chose to fork it. You can frame Amazon as "cringe" and "throwing a tantrum" but ultimately this is hardly either of those.
helen___keller 4y ago
Makes sense if you think about incentives. AWS has a lot of financial incentive to build on open source projects used internally (at a massive scale, I assume). Meanwhile, not a lot of incentive to maintain internal forks on a big project like PostgreSQL
albert_e 4y ago
Is there a good lookup list for all opensource projects and their corresponding AWS branded services? Presto/Trino : Athena
- scientism 4y ago
  It's Azure but, Apache Atlas : MS Purview
up2isomorphism 4y ago
The author seems like a “open source strategist” who does not write code, but if one does, he will likely understand the difficulties of contributing to open source in a cloud service environment and why the observed “free ride”. It will take too many cycles to push an even trivial change upstream and sometimes you can never do that. But your customer does not wait for you, they will happily go to your competitors when they “free ride” more than you do.
EdwardDiego 4y ago
It'd be nice if they open sourced some of their libs. When I was trying to debug a very odd issue with their Python Glue stuff, it was impossible to delve into, because the open source Python just wraps a Java blob you can't see. https://github.com/awslabs/aws-glue-libs/tree/master/awsglue https://github.com/awslabs/aws-glue-libs/tree/master/awsglue
- robertkoss 4y ago
  This is one of the reasons we moved away from Glue.. https://medium.com/claimsforce/lakehouse-running-delta-lake-on-top-of-aws-glue-181133a916f3 https://medium.com/claimsforce/lakehouse-running-delta-lake-...
- JackFr 4y ago
  > just wraps a Java blob you can't see I ahve to admit no familiarity with Glue, but if that Java blob is bytecodes, shouldn't it be trivially decompilable?
  - EdwardDiego 4y ago
    The blob lives in Glue, so not available to decompile.
tracfren 4y ago
I lead client-side application development in AWS IoT and I can tell you we’re investing heavily in open source in AWS IoT. We can build features a lot faster without having to own the infra ourselves and cut out a lot of internal red tape. Customers love features and customization. Examples: AWS IoT Application - https://github.com/awslabs/iot-application https://github.com/awslabs/iot-application - OOB app for monitoring IoT assets. AWS IoT App Kit - https://github.com/awslabs/iot-app-kit https://github.com/awslabs/iot-app-kit - Library for building AWS IoT apps.
throwchainaway 4y ago
To the commenters making embrace/extinguish overtures, I’ll say that log4shell got a lot of Amazon very spooked re. supply chain security. There were a number of internal programs spun up with blank checks to mitigate a future log4shell. The Open Source program was one of those
cs702 4y ago
Demand for a product increases when the prices of its complements decrease. For example, demand for mobile phones increases as the price of mobile OSs and applications declines. To maximize demand for a product, make its complements free. For example, to maximize demand for mobile phones, make the operating system free and add a growing number of built-in default apps for free. Surely, Amazon is thinking along these lines. Which Amazon products/services benefit from all the free code Amazon is contributing/releasing? AWS. See also: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
- m463 4y ago
  I don't see software <-> aws as complements like windows <-> pc, because this software doesn't necessarily need aws like windows
myroon5 4y ago
Author disclosed he worked at AWS, but worth noting he led open source marketing: https://www.linkedin.com/in/mjasay https://www.linkedin.com/in/mjasay
- ctvo 4y ago
  Why is that worth noting other than he probably still has an interest in open source at AWS? He works for a competitor now, which makes this post even more remarkable.
mrtweetyhack 4y ago
[dead]
lamontcg 4y ago
>Though it was always incorrect to lambast AWS for “strip-mining” open source Amazon absolutely strip-mined open source. In 2001-2006 Amazon was all built on open source and had ditched HP and Digital Unix for Linux, RedHat, Apache, Xen, etc. It was abusively hard to contribute back to open source projects, including the Linux kernel, and you had to get extensive signoff and review to ensure that you weren't giving away any IP with your lock order bugfix or whatever. There were also constant reminders and threats to not engage in open source discussion in any way that could leak company IP which created an atmosphere where it was much easier to just not participate at all. It was definitely very easy to pull open source into the company and very, very difficult to participate going the other direction.
- _msw_ 4y ago
  Back when you were at Amazon, I was at Red Hat. (disclosure: I am at Amazon now, and have been for a bit over 13 years). From my perspective when I was at Red Hat, Amazon was very much an IT consumer. Amazon was a _hugely_ important customer for Red Hat, even appearing on stage to give their testimony about the benefits [1]. It was a "lighthouse" customer that really opened some doors to selling Enterprise Software. Part of what we sold to Amazon was being their connection "upstream". In 2001 it was quite unusual for an IT shop like Amazon to participate directly upstream. One benefit of being a customer of Red Hat's was: they will fix bugs and work upstream on your behalf. It was a very reasonable approach in those days to use an existing contractual / support relationship with a vendor like Red Hat to deal with all of those details and complexities, which often required strong social connections among open source developers and maintainers working together upstream. I think that over time as large companies shift to consume open source software directly from upstream, they have to build the capabilities and connections to participate directly rather than going through an intermediary like Red Hat. It can take a fairly long time to do that. [1] https://www.computerworld.com/article/2577912/linuxworld--amazon-com-clicks-with-linux.html https://www.computerworld.com/article/2577912/linuxworld--am...

126 points by unripe_syntax 4y ago

Directionally good, though I'm sure there's a decision tree that's mostly driven by how it benefits AWS. Something like "is committing this back going to save us time maintaining it separately without giving away IP we care about?".

ghaff 4y ago
It's probably not so much IP that they care about as IP that gives them some competitive advantage by keeping it secret. You can probably overstate the degree to which Amazon didn't contribute to and interact with open source communities 5+ years ago (and probably also overstate the degree to which there's been a complete sea change) but, as you say I think it's fair to say there's been a directional shift with more substantive activity in addition to what I'd mostly describe as talking the talk previously. The industry as a whole has also developed into this incredibly complicated coopetition landscape especially around containers in the past decade.
nazgulsenpai 4y ago
I would say that's true for any company that contributes to open source projects.
- tyingq 4y ago
  Yes, though there's some things that are somewhat unique where Amazon is competing with open source products. For example, they have to offer MKS, but would rather you use Kinesis, EventBridge, etc. So perhaps less incentive to make Kafka better for everyone.
jrochkind1 4y ago
I feel like that kind of self-interest is the origin of open source, and always what has powered it when it's working. The original apache httpd, for instance, was written by people working at different organizations collaborating together to build the web server that they needed. It wasn't something people did as just a hobby, although the individual contributors may also have had some altruistic and certainly cooperative motivations. It definitely wasn't something people did motivated by trying to make money off of selling apache httpd in some way. Open source always requires self-interest to fuel it. The original model was people collaborating across organizations because it provided something useful to their organization -- exactly this. When this stops happening, because organizations cut budgets to the bone; because they gatekeep anything that might benefit competitors as well as themselves; because they can free-load off existing open source instead -- what other source of self-interest is there? One is people trying to profit off of the open source itself, with "open core" models, or models where the license can appear open but tries to ensure that the creators maintain a monopoly on any profitable aspects. Or, we can just get lack of investment in open source everyone is trying to free-load off of, withering open source with burnt out maintainers. Companies committing to open source because it saves them the time of maintaining it separately is the best case, and describes the environment of the "golden age" of open source.
- pydry 4y ago
  I met an open source maintainer once working on a very popular tool everybody uses who was practically impoverished. From talking to him it became apparent that he was routinely being taken advantage of across all spheres of his life. It made me wonder how much valuable open source is powered, quietly, by people who are just used to being exploited.
  - pabs3 4y ago
    Probably almost all of it, except large projects with lots of paid devs, like Linux or RedHat.
- taftster 4y ago
  This is insightful. > Open source always requires self-interest to fuel it. That self-interest doesn't necessarily have to be profit motivated either. As one of the replies to your comment suggest, probably the majority of open source is maintained as a hobby by an invested/interested user. Open source does not have to have a profit motive to make sense. But likewise, a profit motive is also congruent with open source. It greases a lot of axles, to put it nicely. There's definitely some disparity in open source contributors in terms of money being made, almost like the income gap between wealth and poverty in general. I wish we could do better at fixing that and helping the "little guy" that is contributing their soul into a project. Still though, open source is driven by "selfish" means, in whatever form that takes.
pkkm 4y ago
Sure, but why does that matter? A lot of good software got developed by itch-scratching.

Dowwie 4y ago

Their s2n projects seem like valuable contributions to the ecosystem. Strong, well maintained libraries supporting networking and cryptography are important.

phpisthebest 4y ago

I wish that same drive would come to their devices division which do the absolutely bear minimum not to be sued for GPL Violation when they use GPL code on their Fire Devices Edit: Also doesn't AWS have a history of Microsoft style EEE, where they embrace an open source project, make a service from it, extend that service in a non-open way and then push people to use their extended non-open service instead of improving upstream? Hasn't this resulted in a number of projects changing their licensing to prevent it?

smatija 4y ago
I think ElasticSearch license change was biggest such change, as was discussed previously on HN: https://news.ycombinator.com/item?id=25776657 https://news.ycombinator.com/item?id=25776657
- pydry 4y ago
  Their "obsession" with contributing isnt really in evidence on their fork: https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da10b894b728b78e77e5e936a6427a7803b2865a0/68747470733a2f2f63646e2e747467746d656469612e636f6d2f726d732f6f6e6c696e65696d616765732f636c6f7564636f6d707574696e672d6769746875625f636f64655f636f6d6d6974732d662e706e67 https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da... Worth bearing in mind that the only reason fork even exists was so that they didnt have to pay Elastic a cut when selling hosted ElasticSearch. If they cared about open source they could start by not undermining the business models of companies that do more than just pretend to support it.
  - TravelPiglet 4y ago
    Having to pay for hosting an open source product? Isn’t Amazon doing what they should in this case where a company tries to block any competition.
    - pydry 4y ago
      No. Amazon was abusing their enormous market power to squeeze the company that made this software out of the market of hosting it. (Not to mention lying about "partnering" with them) If allowed to continue this would kill the project. If only Amazon can monetize it, it doesnt get developed. The open source project then dies. Amazon is no more doing the right thing here than they were when they destroyed diapers.com with a strategy of sustained predatory pricing. The fact that they want a pat on the back for being "pro open source" now after pulling this kind of crap is just icing on the cake.

andrewstuart 4y ago

One of the things that led me to lose interest in AWS was its attitude to open source.

pydry 4y ago

>Though it was always incorrect to lambast AWS for “strip-mining” open source It really wasn't. That was their whole approach to Elastic. The tantrum they threw at Elastic when they relicensed was particularly cringeworthy.

prepend 4y ago
They didn’t throw a tantrum. They just forked and continued. A perfectly reasonable approach. I think it’s important to describe things accurately. Disliking something isn’t a “tantrum.” I think amazons blog posts were pretty normal speech and tl:dr’d to “we don’t like it, so now we forked it and will maintain under Apache.” And they did that, so it’s actually kind of nice as I can’t use ElasticSearch under its new license but can use open search.
- pydry 4y ago
  Tantrum is pretty accurate I think. They voiced their extreme displeasure in a war of words with Elastic. They knew that the relicensing was targeted 100% at their gravy train and they condescendingly declared that they would "step up on behalf of open source" while they stepped up exclusively to prevent Elastic getting a cut of their ElasticSearch hosting profits. The fork is really half hearted with about 1/10th the resources Elastic dedicates to the mainline branch: https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da... Meanwhile, the only effective licensing difference is that you can sell hosting with their fork - something that only benefits them.
  - belval 4y ago
    A lot of forks are created for similar reasons, it's not a fight between the great enlightened and the vile defilers. It's a for-profit (Elastic) wanting more control over their product (ElasticSearch) while also making ElasticSearch open-source under a permissive license that they used to kick-start their growth and another for-profit (Amazon) leveraging the open-source license to offer the product as a service to their customer base. When they had a disagreement Amazon chose to fork it. You can frame Amazon as "cringe" and "throwing a tantrum" but ultimately this is hardly either of those.

helen___keller 4y ago

Makes sense if you think about incentives. AWS has a lot of financial incentive to build on open source projects used internally (at a massive scale, I assume). Meanwhile, not a lot of incentive to maintain internal forks on a big project like PostgreSQL

albert_e 4y ago

Is there a good lookup list for all opensource projects and their corresponding AWS branded services? Presto/Trino : Athena

scientism 4y ago
It's Azure but, Apache Atlas : MS Purview

up2isomorphism 4y ago

The author seems like a “open source strategist” who does not write code, but if one does, he will likely understand the difficulties of contributing to open source in a cloud service environment and why the observed “free ride”. It will take too many cycles to push an even trivial change upstream and sometimes you can never do that. But your customer does not wait for you, they will happily go to your competitors when they “free ride” more than you do.

EdwardDiego 4y ago

It'd be nice if they open sourced some of their libs. When I was trying to debug a very odd issue with their Python Glue stuff, it was impossible to delve into, because the open source Python just wraps a Java blob you can't see. https://github.com/awslabs/aws-glue-libs/tree/master/awsglue https://github.com/awslabs/aws-glue-libs/tree/master/awsglue

robertkoss 4y ago
This is one of the reasons we moved away from Glue.. https://medium.com/claimsforce/lakehouse-running-delta-lake-on-top-of-aws-glue-181133a916f3 https://medium.com/claimsforce/lakehouse-running-delta-lake-...
JackFr 4y ago
> just wraps a Java blob you can't see I ahve to admit no familiarity with Glue, but if that Java blob is bytecodes, shouldn't it be trivially decompilable?
- EdwardDiego 4y ago
  The blob lives in Glue, so not available to decompile.

tracfren 4y ago

I lead client-side application development in AWS IoT and I can tell you we’re investing heavily in open source in AWS IoT. We can build features a lot faster without having to own the infra ourselves and cut out a lot of internal red tape. Customers love features and customization. Examples: AWS IoT Application - https://github.com/awslabs/iot-application https://github.com/awslabs/iot-application - OOB app for monitoring IoT assets. AWS IoT App Kit - https://github.com/awslabs/iot-app-kit https://github.com/awslabs/iot-app-kit - Library for building AWS IoT apps.

throwchainaway 4y ago

To the commenters making embrace/extinguish overtures, I’ll say that log4shell got a lot of Amazon very spooked re. supply chain security. There were a number of internal programs spun up with blank checks to mitigate a future log4shell. The Open Source program was one of those

cs702 4y ago

Demand for a product increases when the prices of its complements decrease. For example, demand for mobile phones increases as the price of mobile OSs and applications declines. To maximize demand for a product, make its complements free. For example, to maximize demand for mobile phones, make the operating system free and add a growing number of built-in default apps for free. Surely, Amazon is thinking along these lines. Which Amazon products/services benefit from all the free code Amazon is contributing/releasing? AWS. See also: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

m463 4y ago
I don't see software <-> aws as complements like windows <-> pc, because this software doesn't necessarily need aws like windows

myroon5 4y ago

Author disclosed he worked at AWS, but worth noting he led open source marketing: https://www.linkedin.com/in/mjasay https://www.linkedin.com/in/mjasay

ctvo 4y ago
Why is that worth noting other than he probably still has an interest in open source at AWS? He works for a competitor now, which makes this post even more remarkable.

mrtweetyhack 4y ago

[dead]

lamontcg 4y ago

>Though it was always incorrect to lambast AWS for “strip-mining” open source Amazon absolutely strip-mined open source. In 2001-2006 Amazon was all built on open source and had ditched HP and Digital Unix for Linux, RedHat, Apache, Xen, etc. It was abusively hard to contribute back to open source projects, including the Linux kernel, and you had to get extensive signoff and review to ensure that you weren't giving away any IP with your lock order bugfix or whatever. There were also constant reminders and threats to not engage in open source discussion in any way that could leak company IP which created an atmosphere where it was much easier to just not participate at all. It was definitely very easy to pull open source into the company and very, very difficult to participate going the other direction.

_msw_ 4y ago
Back when you were at Amazon, I was at Red Hat. (disclosure: I am at Amazon now, and have been for a bit over 13 years). From my perspective when I was at Red Hat, Amazon was very much an IT consumer. Amazon was a _hugely_ important customer for Red Hat, even appearing on stage to give their testimony about the benefits [1]. It was a "lighthouse" customer that really opened some doors to selling Enterprise Software. Part of what we sold to Amazon was being their connection "upstream". In 2001 it was quite unusual for an IT shop like Amazon to participate directly upstream. One benefit of being a customer of Red Hat's was: they will fix bugs and work upstream on your behalf. It was a very reasonable approach in those days to use an existing contractual / support relationship with a vendor like Red Hat to deal with all of those details and complexities, which often required strong social connections among open source developers and maintainers working together upstream. I think that over time as large companies shift to consume open source software directly from upstream, they have to build the capabilities and connections to participate directly rather than going through an intermediary like Red Hat. It can take a fairly long time to do that. [1] https://www.computerworld.com/article/2577912/linuxworld--amazon-com-clicks-with-linux.html https://www.computerworld.com/article/2577912/linuxworld--am...