20 ms·

It's hard for me to know whether to feel bad for ES in this case. Did they bring it on themselves? Is Amazon too big and a bully? From my perspective, Amazon h

It's hard for me to know whether to feel bad for ES in this case. Did they bring it on themselves? Is Amazon too big and a bully? From my perspective, Amazon has made most of its profit price gouging consumers on bandwidth after vendor locking them into their ecosystem, where they bootstrap new services by wrapping open source software with some provisioning scripts, management dashboards and cookie-cutter API / console templates. Indeed, most of this is templated -- AFAIU, for example, each AWS service autogenerates its Boto bindings and parts of its console frontend via code generators. Amazon has really mastered the factory process of churning out new services, and when they find a popular one, they can invest more resources into developing it than the original team ever could. And therein lies the rub. If Amazon is improving the software in a way that the original team couldn't, it's hard to say that the community isn't benefiting. I think what strikes me the wrong way is that Amazon is not doing it for any altruistic reason. In fact, Amazon contributes very little to open source in general, considering how much they take from it. Compare them to Facebook (React, etc) or Google (tons of dev tools) or Microsoft (VSC, TypeScript). What does Amazon have? Firecracker, kind of? And now a fork of ES because that's the only way they could continue making money off it without violating the license a small startup put in place to stop them? Well, good for Amazon, I suppose, but I find myself instinctively disliking them for this. I'm not sure what the solution is. Hopefully technologies like Kubernetes and Terraform will encourage big customers to become at least cloud-agnostic, if not cloud-independent. At the very least it would be great if Amazon / Google / Microsoft stopped gouging bandwidth at such absurd margins. Or not. Maybe it will be their downfall as startups differentiate along those lines. That would be ironic, coming from the originators of "your margin is my opportunity." Personally I'm doing my part by not building anything with vendor lock-in. It's great to be able to deploy to any cloud, if you value either robustness or flexibility.

jnwatson 6y ago
How is it price gouging if the price is on the tin? It isn't like there's a "surprise" as to how much they charge, and it isn't like there aren't a dozen alternatives including DIY. I'm the first to say that AWS is too expensive, and I vote with my wallet (and the company I work for by proxy). But I'll never claim that there's any gouging involved.
- ballenf 6y ago
  Price gouging is the practice of using outsized leverage in a particular market to charge excessive prices. Like snow shovels doubling in price after a snow storm. Or $10 water bottles after a hurricane. So for AWS the term is arguably correctly applied. But I'd be more worried about the market if AWS was artificially undercutting pricing because it would kill the incentive to create competitors or innovation in the space.
  - sofixa 6y ago
    > Price gouging is the practice of using outsized leverage in a particular market to charge excessive prices. Like snow shovels doubling in price after a snow storm. Or $10 water bottles after a hurricane. > So for AWS the term is arguably correctly applied What outsized leverage have AWS had for a decade? There are multiple competitors at different levels, AWS are just better in terms of coverage/redundancy and amount of services.
    - pas 6y ago
      They are meaningfully better only in one thing: size = PR = network effects.
  - jen20 6y ago
    > So for AWS the term is arguably correctly applied. Can you elicit the argument by which this is correctly applied?
    - ballenf 6y ago
      Point was just that having the "price on the tin" is irrelevant to whether there's price gouging going on. But, I guess the argument could be similar to the case against Apple for its iOS App Store: there are lots of competitors, but the lock-in arguably creates a market definition of Apple customers. AWS customers are largely locked-in and at the mercy of AWS prices. That's the argument. I'm not sure I buy it, but it's one perspective.
busterarm 6y ago
As a longtime ElasticSearch cluster admin/developer and Elastic Cloud customer, I don't feel bad for Elastic in the slightest and I'm psyched about this fork. The way they operate their cloud service leaves a lot to be desired and encourages maximum spend if you end up wanting to use it for anything demanding in production.
- cassianoleal 6y ago
  Not to mention the support is pretty slow to respond and rarely respond adequately. At my current client we started with their cloud and in a few months deployed our own in Kubernetes using the official operator. It's a lot cheaper and gives us fewer headaches this way.
  - busterarm 6y ago
    My favorite moment as a customer dealing with all of the random times _they_ would take us down (even with triple redundancy) was their suggestion of operating a duplicate 3 datacenter cluster in production as a hot spare. They must think their customers are cash machines.
    - bombcar 6y ago
      I mean if they were truely best of breed they wouldn’t have needed a license change now would they?
      - busterarm 6y ago
        Indeed, although I'd say for a majority of situations, Amazon's hosted Elasticsearch service is a complete non-starter.
        damon_c 6y ago
        Why is it a non-starter? I have a few ElasticSearch clusters that have been running for years with zero effort/downtime/hassle.
fastball 6y ago
I think the claim that Amazon is winning through "vendor lock-in" is pretty silly. Honestly anyone who can't quickly migrate the stuff they're hosting on AWS onto one of the many other cloud platforms is pretty bad at DevOps. If you're using K8S/Docker/etc it should be trivial. But even if you're not, the vast majority of AWS offerings were either built to be API-compatible with other existing tools (e.g. postgres-compat Aurora RDS), are literally identical to other services you can self-host (e.g. ElasticSearch) or others have built services compatible with AWS services (e.g. DigitalOcean's "Spaces" aim to be API identical to AWS S3 – you can literally use Amazon's S3 client libs to interact with various S3-compat services from other clouds). It's not "lock-in", it's providing a great all-in-one solution. You can host everything you want to host on AWS, which has good stability, good latency, etc. People are locked in because the DX around using AWS for everything your platform needs is just better than other platforms / having different services on different cloud providers (at least for many people).
- real_joschi 6y ago
  If you're not using any of the AWS services, that might be true but then you're also leaving a lot of potential on the table. If you're "cloud-agnostic" and could migrate away from AWS in the blink of an eye then you're paying for an overpriced VM offering and should probably migrate to a cheaper hosting provider immediately.
  - busterarm 6y ago
    > If you're "cloud-agnostic" and could migrate away from AWS in the blink of an eye then you're paying for an overpriced VM offering and should probably migrate to a cheaper hosting provider immediately. No this is what everyone suggesting this does not get. The offerings are not equivalent. There are a baseline of services that the cloud providers offer that can be made functionally-equivalent. It's not just EC2 but more like EC2,S3,Lambda,RDS,DynamoDB,ECS,EKS (plus some others and of course the other cloud's equivalents). The secret sauce is in the APIs and permisioning and all of these available within the same VPC (talking to each other without paying bandwidth costs). "Cloud-agnostic" has _never_ meant "just VMs". Some of these services are majorly hard to duplicate on your own VMs as well. Feel free to implement "cheap VM hosting + S3" and burn cash on transit costs. Cheaper hosting providers do not give you this by miles.
minhazm 6y ago
Is it really price gouging for bandwidth? Or is bandwidth just really expensive in general? I honestly don't know. I would assume if it was actually much cheaper one of the cloud's would undercut the other to get customers.
- GordonS 6y ago
  It really is price gouging - bandwidth is actually cheap. A couple of comparisons: Oracle Cloud give you 10TB of bandwidth for free, with overage charged at around €7.5/TB. You can rent a VPS from the likes of Hetzner, and they will throw in 2-20TB of bandwidth for free, with overage charged at something like €1/TB - AWS charge an eye-watering €125 for each TB! I think the reason the big 3 (AWS, Azure, GCP) still charge such huge amounts is that they profit so greatly from it, and there is more than enough business to go round.
  - martinald 6y ago
    I totally agree, but was stunned to see DigitalOcean charging $0.10/GB for outbound transfer for their new (quite cool imo) apps service. You do get 40GB-100GB included, but it means it's unusable for bandwidth heavy apps. They include 1TB (which is pooled across all VMs) and 0.01/GB on their standard platform.
    - GordonS 6y ago
      Yeah, I think Digital Ocean are really taking the piss there - proven by their existing $0.01/GB pricing, as you mentioned. I reckon DO have seen the big providers getting away with it, and they want a piece. Why wouldn't they, when there is apparently so little pushback?
      - martinald 6y ago
        I suppose you have object storage which is still $0.01/GB (plus includes 1TB to start with), so for most apps your bandwidth will come out of that for most files, so your 0.1/GB price is only for html/api/etc transfer. But still, it's annoying and seems completely arbitatary.
deleted 6y ago
[deleted]
notyourday 6y ago
As someone who tried buying services from ES and had to deal with their smug sales people that had a total disdain to those who wanted to give ES money, I am happy I will never need to deal with them in future.
prepend 6y ago
> In fact, Amazon contributes very little to open source in general, considering how much they take from it I don’t think this is a fact. Amazon seems to contribute pretty significantly according to the pages [0,1] they put out that describes their contributions. Not to mention their membership in OSS foundations like Linux Foundation. [2] You have the caveat about in relation to benefit they gain, but that’s pretty hard to measure. And I think isn’t really a good measure. I’d like to learn more about why you make such an absolute claim and maybe you have some better measure. I remember back in the 90s when big orgs (Microsoft, IBM) didn’t contribute to open source and can’t even think of any big orgs today that don’t contribute to open source. Even Oracle has big open source projects. [0] https://amzn.github.io/ https://amzn.github.io/ [1] https://aws.amazon.com/opensource/ https://aws.amazon.com/opensource/ [2] https://www.linuxfoundation.org/en/join/members/ https://www.linuxfoundation.org/en/join/members/
- cbushko 6y ago
  To me it looks like they mostly contribute to projects that are SDKs to use AWS.
- aledalgrande 6y ago
  Yeah and also what about projects like Firecracker? https://github.com/firecracker-microvm/firecracker/ https://github.com/firecracker-microvm/firecracker/
- TechBro8615 6y ago
  The absolute claim, aside from being at home in a rant on HN, comes from a cursory glance at https://github.com/amzn https://github.com/amzn, weighted by contributors and popularity, and compared to companies of similar size. Google, Microsoft and Facebook all build and maintain multiple open source projects that are hugely popular with people who use them outside of the company sandboxes. For example, people benefit from React without Facebook gaining much directly. (Facebook! If Facebook has any redeeming qualities, it's their open source contributions to the frontend ecosystem, although I promise you I could ascribe malicious intent to those as well...) Contrast that with Amazon. On their GitHub page, I see a few obscure projects amongst a bulbous array of AWS SDKs. To the sibling comment that asked about Firecracker -- I think Firecracker is awesome, and I did mention that in my original complaint. They even created it themselves! Well, a team of amazing engineers in Romania did. I have no personal insight into the matter, but it seems like they operate relatively independently from the AWS profit machine. Good for them too, it's incredible software. But I'm sure if they were to tell the story of how they got buy-in at Amazon to open source it, the same themes would come up -- how does Amazon benefit from this? In the case of Firecracker, the more people test it / harden it / run Doom on it, the more value Amazon can provide on its serverless platform. So again, unlikely to be purely altruistic intentions... but that's not to say there's anything wrong with that. I just find it all a bit distasteful in aggregate.
z77dj3kl 6y ago
My favorite "conspiracy theory" is that AWS intentionally creates stupidly verbose and numerous headers in all of their APIs just to up the bandwidth usage a few bytes per request at a time.
- deleted 6y ago
  [deleted]
- cortesoft 6y ago
  You don’t pay for the bandwidth of API calls to AWS
015a 6y ago
One thing which surprised me: Elastic has a market capitalization of ~$11B. I think that changes some of the more floaty ethical concerns. This is not a David vs Goliath situation. This is Goliath vs Super-Goliath. At this point, I'm much less interested in the drama of which mega-corp is screwing over the other. I'm more interested in: how does it affect me? When the titans are done trampling over the rest of us, which side benefits me the most? Its too early to tell, but it seems like it'll be Amazon. The product is more open. They have a demonstrated history of great support. Yeah, they gouge us on networking and everything else, but at least they're the devil we know, and buying into the OpenSearch ecosystem has a greater probability of being the more open solution into the next decade.
- aardvarkr 6y ago
  Uhmmm I’m pretty sure David vs Goliath is talking about scale between competition. Saying that $11B is Goliath just because you’re sitting at $1M doesn’t mean they’re not in a crazy mismatched fight against a $2T company. In the same way you could be in a David vs Goliath situation yourself if you with $1M in wealth tried to sue someone with $25K of wealth. Everything is relative. Doesn’t mean it’s not a crazy unfair mismatch that doesn’t deserve sympathy and regret.
  - bombcar 6y ago
    When there are two large entities (and I take this to mean way larger than you or your company) then you’re better off rooting for the one that at least releases code you can use. If OpenSearch is truely open then in theory you can find another provider. But for ElasticSearch you’re stuck with them.
    - kfir 6y ago
      > But for ElasticSearch you’re stuck with them. I am a bit at a loss about that statement, you can run Elasticsearch on your on infrastructure
      - virgilp 6y ago
gsich 6y ago
No need to feel bad. The CEO sold a lot of his shares prior to the license change.
ampdepolymerase 6y ago
What's worse is the second order repercussions. Future open core/open source SaaSes will go straight to something like the Business Source License or the MongoDB license instead of traditional libre licenses. Amazon has done an incredible amount of damage to the open ecosystem.
- randoramax 6y ago
  Good. So the VCs will go back to playing like in the old days of shareware software licenses. It'll be good.
hintymad 6y ago
It sounds trivial to "wrap open source software", but surprisingly it is big value-add to thousands of companies. We can't just look at successful companies like Netflix to downplay the challenges of operating a service. Not every company knows how to operate complex systems under manageable cost. How many companies can really manage a Kafka cluster, let alone scaling it, for instance? Indeed, even companies that people deem powerful may screw up, if they don't get their culture or process right. Take Uber for example, for god damn five years, they still couldn't offer a service like EC2, let alone supporting persistent volumes. They still couldn't make their database provisioning on demand via an API. Their MySQL-based NoSQL solution was still based on FriendFeed's architecture and the APIs were hard to use. Yet they spent millions building a k8s replacement, building a GPU database, switching from mysql to postgres and back to mysql, etc and etc. So, yes, cloud companies like AWS buildd mere control planes to wrap open-source software, yet such seemingly mundane offering does bring values to many customers.
- cle 6y ago
  This is the most important point, IMO. Amazon's value add is not the software itself, it's the operation of the software. That includes a LOT of stuff, not just making sure it's running. It's security modeling and patching, compliance, DDOS protection, etc. Amazon's product is an army of ops engineers working 24/7 to keep your stuff secure and online. With that in mind, their behavior here makes a lot more sense, and comparing it with companies who have dramatically different products, like Facebook and Google, takes a lot of effort to understand the differences and what impact they have.
  - harshaw 6y ago
    AWS just has engineers full stop. No specific ops engineers.
  - stefan_ 6y ago
    You can rent a car but nobody would suggest the car manufacturer be paid nothing for the privilege just cause the rental company cracked car maintenance and how to fill the tank. The AWS "value add" is only value add in the context of being locked into AWS in the first place.
throwaway823882 6y ago
> Is Amazon too big and a bully? Is Starbucks too big and a bully? Sure, they force Mom and Pop shops to close by out-competing them, and that sucks. But bullying? I believe that's just Capitalism. Or say you're 6'6 and weigh 230LBS and you join a football team full of people who are 5'9 and 175lbs. Are you a bully just because you're bigger? ES basically handed them a platter with a goose laying golden eggs and a sign that read "Free Goose" and hoped they wouldn't try to make money off the eggs.
- StavrosK 6y ago
  > Is Starbucks too big and a bully? Sure, they force Mom and Pop shops to close by out-competing them, and that sucks. But bullying? I believe that's just Capitalism. I'd say it's more like a mom-and-pop coffee shop giving free coffee to patrons hoping to make money on cookies, and Starbucks coming in, taking the free coffee, opening a nice stand right next to the shop and selling the coffee they got for free. I'm having trouble justifying this ethically.
  - throwaway823882 6y ago
    If they're giving you free coffee, and you sell the coffee you got for free... where's the harm? Starbucks is giving it away. Why would anyone buy your coffee rather than get it for free from the source? Now, assuming Starbucks charged you for the coffee, and you then re-sold it, this should also be fine. Starbucks is charging you presumably a rate with which they can recoup their costs. But if they are selling it below cost, they are clearly putting themselves at risk. A lot of businesses take this kind of risk as a strategic part of their business, like with making the coffee free. But you have to do it in a way that a competitor can't turn around to their advantage, or you're screwing yourself. Enter the concept of "not for resale". If a seller enters into a contract with a buyer, that contract can stipulate that the buyer can't resell the goods. Starbucks could theoretically require you to sign a contract saying you will not resell their coffee. That's pretty standard with licenses, even software licenses. ES must have known that their license did not forbid reselling. Yet they based their business model on this resellable coffee that they were giving away in order to make money on cookies. Is it Amazon's fault that ES chose a business model where they were selling coffee at a loss? Does Amazon have an ethical responsibility to keep ES's business afloat? Should we find any business unethical that tries to undercut the customer base of a rival, or take advantage of a rival's shaky business model? I think you have to come up with a whole framework for ethical competition, because one rule at a time isn't gonna capture it. (But I also think Capitalism is inherently unethical)
jrochkind1 6y ago
What ES wants of course is for Amazon to give them a cut of revenue from hosting ES. We already know Amazon isn't interested in doing that (either at all, or at whatever price ES wanted, we don't know that). They had no legal requirement to when ES was open source. So ES changed the licensing to no longer be open source. So, Amazon could... a) decide to give ES a cut after all, b) decide to stop hosting ES, or c) fork the last open source version. I don't think anyone is surprised they chose c? Presumably ES isn't either? Maybe ES thinks this will be good for them/bad for Amazon anyway, because they are hoping potential customers will abandon the Amazon fork and stay on the original ES fork? Not sure why they'd be confident in that exactly. Maybe they know what they're doing. As users/customers, we would rather have a choice of hosted vendors/platforms, and that it remain un-forked (so we can use/write software compatible with either vendor/platform). Competition is good for us as users/customers, that's in fact one of the reasons we choose open source, so no one vendor can set the hosting price all on their own without competition. We want to be able to choose among competitors for hosting, based on price, customer service, performance, uptime, whatever. But ES didn't want that, they didn't want hosting competition to exist -- at least not without permission and agreed upon cut for them -- because, I guess, hosting was how they planned to make money as a company to fund development as well as profits for investors etc. So they changed their license to no longer allow it. So of the possible outcomes remaining... this one seems as good as any for the user/customer, I guess? So, when you say "I'm doing my part by not building anything with vendor lock-in" -- I'm not sure which course you are suggesting. In fact, between ElasticSearch and new OpenSearch fork.. it's OpenSearch that is the one without vendor lock-in, right? OpenSearch is Apache licensed, and can be hosted by any vendor and still forked by anyone . It's ElasticSearch that has a license limiting what vendors can host it (without permission of ES), it's the one with vendor lock-in, right? So not building anything with vendor lock-in means... ?
- galaxyLogic 6y ago
  Good interesting points. Now Amazon will be the good guy because they will run open-source version, whereas ElasticSearch is not, if I understand you correctly. No single capitalist wants competitive markets. They want monopoly, for themselves. It is only when they don't have the monopoly or an easy way to get it that they cry for competitive markets. And that is good of course.
tssva 6y ago
"without violating the license a small startup put in place to stop them?" Elasticsearch was first released over a decade ago. ElasticSearch, now just Elastic, the company was founded over 9 years ago and now is public. Are they still a "small startup"? If so when does a company graduate from that status?
manigandham 6y ago
People want to pay for services, not software and licenses. They want turn-key solutions that are available via API and GUI, instantly and on-demand. This is the fundamental reason why AWS is so successful and the demand is constantly proven with every new product launch. Elastic (and other vendors) complaining about this instead of using it for their own success is a problem of their own making. At least a few companies are finally learning.
reflexe 6y ago
It it quite funny, elasticsearch is also kind of wrapping an open source library (Apache Lucene) and selling it as their own product.
- jrochkind1 6y ago
  Very good point. Could Elastic's business model survive if lucene adopted the SSPL license that Elastic has, saying it's the "spirit" of open source? That is an interesting question.
  - pas 6y ago
    The problem is, that AWS is not really sharing the internals of their fork. Big corps always say that there's no point in open sourcing their fork, because it depends so much on internal systems. Which might be true, but in case of AWS and infrastructure the possibility of an AWS competitor similarly implementing those internal services would probably help the market... MySQL for example took a proprietary API (the mSQL API) and implemented an open software for it. Then Oracle bought MySQL. MariaDB started and it quickly dropped full compatibility. Options for the developers/users. Great. AWS/GCP/Azure on the other hand distorts the market. Just as Google/Apple/MS fund Chrome/Safari/Edge development from other sources, thereby distorting the market, forcing browsers to be free as in beer. (Which arguably hurts the market.)
    - jrochkind1 6y ago
      The announcement says "All of the software in the OpenSearch project is released under the Apache License, Version 2.0 (ALv2)." You are suggesting this is not correct?
scarface74 6y ago
There is absolutely nothing “cloud agnostic” about using Terraform. Every provisioner is specific to a cloud provider. If you are at any scale, moving a k8s cluster is the least of your issues.
dieters 6y ago
There are also ways out of vendor lock-in. Alternator comes to mind as a way of migrating DynamoDB workloads out to other cloud vendors or your own servers: https://docs.scylladb.com/using-scylla/alternator/ https://docs.scylladb.com/using-scylla/alternator/
dariusj18 6y ago
> Amazon is not doing it for any altruistic reason The beauty of OSS is that motives don't matter. If Amazon contributes and it's not detrimental in someway to the code, then it's a plus for anyone else who wants to use it.
- DetroitThrow 6y ago
  Precisely! While their business itself may need to be broken up, a community governed OSS project isn't bad for OSS when the alternative is a proprietary license that gives a single corp the ability to not contribute back or be exposed to virality. All this being said, progressive corporate taxes seem more enticing year after year.
raiflip 6y ago
It seems reductionist to say Amazon primarily wraps around open source. What about EC2? S3? Glue? DynamoDB? Many of the services that provide the most value are services Amazon has built out.
- sofixa 6y ago
  Om top of these, many of the core services that AWS themselves rely on, like SQS, SNS, Kinesis, Lambda, Cloudfront, ECS, Fargate, Elastic Beanstalk are mostly homemade
- Nemo_bis 6y ago
  EC2 uses the KVM hypervisor.
amzn-throw 6y ago
> they can invest more resources into developing it than the original team ever could. I know this is a popular narrative, but as someone who works on AWS, I think you would be shocked by how small the individual dev teams are that build and maintain the services that everyone uses. I'm not going to downplay the network effects involved. Of course AWS has a tremendous advantage in being able to standardize the customer billing, IAM, and EC2 Usage. And there are economies of scale. But individual AWS service teams are: * incredibly lean and focused * still have to make a profit on their own terms based on the infrastructure they build and the fees they charge customers * laser customer obsessed to solve people's (developer's) direct needs. I understand the community's concern about AWS investment and approach to OSS. But I can assure you (though you have no reason to believe me) that the goal is never to embrace, extend, then extinguish. It's all in the service of going where the customers are, and solving problems that they tell us they have. The profits are a byproduct. The "working backwards" process is no joke. We spend a lot of time figuring out what is the right thing for customers to build, start building it, and THEN we think about how do we make money from it.
- waynesonfire 6y ago
  they have as many people dedicated to the capability as needed. at least the capabilities have owners, which from my experience is not trivial for an organization to achieve. curious if others have noticed this as well (capabilities without clear owners)? what does this depend on? time? company size? both?
- g9yuayon 6y ago
  In the world of OSS, code speaks. A small team may not help the cause of Open Search. This article nails how code influences leadership in an OSS project: http://hypercritical.co/2013/04/12/code-hard-or-go-home http://hypercritical.co/2013/04/12/code-hard-or-go-home
- systemvoltage 6y ago
  Could you please shed some light on how many people would be behind a product like AWS Lambda or AWS CloudWatch? As an outsider, I would guess huge swaths of developers with a massive hierarchy. Buildings full of folks working on AWS services. I have no idea and extremely curious.
grumple 6y ago
Here are the main reasons to make open source software: To provide something for free, public use To get the world to help you maintain your software A reason to not make open source software: You want the exclusive right to offer that software as part of a paid service This is not the first instance of a company not understanding the last point there, and it won't be the last.
downrightmike 6y ago
This is Amazon's playbook. Make a direct competitor and squeeze the originals out. They did that with jewelry early on and then anyone they couldn't buy out they would under cut until they capitulated like diapers.com The Everything Book by Brad Stone goes over this in detail. Clearly anti-competitive monopolistic actions are taken constantly by Amazon. The only reason they aren't trust busted is because the common line of reasoning is that consumers pay less for goods, but this is being looked at because shouldn't competition be lowering prices. IE if Amazon hadn't killed diapers.com, wouldn't diapers be cheaper overall? And the answer is they should be, but the government hasn't caught up. Once they start getting into the weeds, they'll see example after example of monopoly behavior destroying competitors and ultimately raising prices on consumers.
marvindanig 6y ago
Coincidentally, AWS hasn't open-sourced anything that they use internally. Zilch. Nada. And yet they are using 'open source' developed by another firm (smaller is inconsequential here) to market themselves. IMO, FOSS licensing is completely broken. Its definitions (of what is free/open) are from a boomer's era that is no longer sustainable. At least I wouldn’t want any of my FOSS projects become “corporate strategy” of any particular proto-nation.
- pas 6y ago
  That made me wonder which megacorp did? Google published a lot of papers about their architecture, but no code. They have many open source projects modeled after internal tools (eg. bazel), but there's no search/GFS/mapreduce/monorepo OSS project from them. MS has VSCode, that's great, and they even open sourced .NET. But Azure is a full black box, just like (almost) every Windows component. (Finally calc.exe is OSS!)
- PHGamer 6y ago
  gplv3 was supposed to fix this though no?
- zokier 6y ago
  > Coincidentally, AWS hasn't open-sourced anything that they use internally. Zilch. Nada. Firecracker. s2n. Not much, I agree
api 6y ago
You described the cloud lock in model very well, especially the bandwidth part. What they charge for outbound is nuts, and the other clouds are not much better. Inbound is free of course to make it easy to send your data in but costly to get it out.
brutal_chaos_ 6y ago
Did the BSDs do it wrong, too? Apple uses a lot of FreeBSD software that they turn around and sell for profit. How about PostgreSQL, didn't Amazon fork that as well? My point is, there's nothing wrong with forks nor companies forking if the license allows for it. It's up to the developers to choose an appropriate license to not be forked/ripped off, if they so desire. I personally am against modern day corporate America, but I can't blame them for this. The software is given away free/libre/gratis to be forked by whomever. Perhaps to combat this, one should choose a non "Open Source(TM)" license, but a source availbe license. E.g https://mariadb.com/bsl11/ https://mariadb.com/bsl11/ (not my personal favorite, just an example). Also, I definitely agree with/do the same: > Personally I'm doing my part by not building anything with vendor lock-in. It's great to be able to deploy to any cloud, if you value either robustness or flexibility.
JoshTriplett 6y ago
When a product that was previously Open Source changes to a non-open license, it's not uncommon for someone to pick up the last Open Source version and fork it, and release that for others to use and collaborate on. That's always going to be a good thing; that means people who care about Open Source licensing continue to have a version to use and collaborate on.
jakear 6y ago
The main thing I look at in this situation is their approaches to security. ES decided authentication is a paid-only feature implemented via closed source proprietary code, and the result has been countless PII leaks; now sure you could say that’s the developers’ fault for making the endpoints internet-accessible, but when the system has been designed from the start to both be insecure and hold PII, you have to place some blame on the provider as well. Amazon on the other hand developed a free and open source auth plugin and anyone is able to deploy it no problem.
killingtime74 6y ago
Only good one I can think of is Amazon Java, not sure how essential it is though
bitonp 6y ago
I am with you. The release could read: "having subsumed the opensource version of ES, we are now relicensing, calling it our own, and would be really happy if the opensource community would lie to contribute, because actually we don't totally understand how this product works. Many thanks to all who help us" The issue with AWS version are many fold, but the main one is that it forces extra usage of expensive EC2 units, for the following reasons: 1. Blue/Green updates -> Start a new cluster with new version, lock current cluster, copy over all the data (can take over a day), at the same time write to both clusters, when finished, lock both clusters while endpoints are swapped over, unlock new cluster, trash old cluster. During this process Customer pays for both. - Solution, done properly, fire up new node with new version, swap it in, wait for 'green', take out old... wait for 'green'.. rinse and repeat. Result.. system never goes down, endpoints remain the same, less cost to customer. 2.There are a minimum of the folloowing node types: - Master (small) x3 - Data (heavy on storage, medium on memory) - Hot Data (very heavy on memory as shards have to be held in memory) - Coordinating (query) nodes, heavy on memory, light on storage (cos there is no real storage) - Ingest (same as Query). - voting only.. tiny - AI/ML heavy on memory and storage.. cos they do real work In the AS world they have: - Masters - Data - Hot Data (cos they are really pricey) The Data nodes do all teh functionality of the data, ingest and query nodes. Th emain query always gores to a data node, while it passes to other data nodes to get the data out, then aggregates locally to it... so its incorrect usage of the system. A data node should only ever deal with its own data, and pass the results it finds back to a node away from the data. As your system is squeezed.. you add data nodes... not coordinating/Query or Ingest nodes, which you probably need. But thats more money into the coffers. 3. Their userhasging function is also old (2011 vesrion of bcrypt) , and fails. Any static password produces different results everytime you use it.. at least on the current opendistro. So you are forced into either not using security, or using proper security, which can be cumbersome across the cluster (if rolling your own). however.. on the cloud version they have base level security working, so thats Ok.. they arent using their own 'open' software. I could go on.. but its all flaky, and misunderstood at the core developer level. Toput on record, I have spoken twice to teh core development team to describe how updates should happen.. the last time 18 months ago (by 'spoken' I mean a face to face video call) .. so they know hoe it should be done.. but dont do it. 4. I have noticed that there are some functions/methods available on the setup in Amazon Linux, that dont exist on other Linux versions (centOs/Debian) that are security related. AWS Linux is 'lifted' from Redhat.. so another piece of software that they didnt write.. but obviously Redhat are happy with this. Maybe they got the licensing deal sorted .. who knows. basically gores like this: "Who would actually install, in production, what essentially is very close to pirated software?" Roll your own.. its a bit of grunt work up front, then 50% the cost of the cloud version.
llcoolv 6y ago
So VSCode, which is basically slomo Sublime Text for retards and Typescript which is usable only for prototyping are a good enough compensation for having most of their income from Linux (on Win and Azure) after all the almost criminal FUD they poured on Linux[0]? Come on, get real. 0. https://en.wikipedia.org/wiki/Halloween_documents https://en.wikipedia.org/wiki/Halloween_documents

jnwatson 6y ago

How is it price gouging if the price is on the tin? It isn't like there's a "surprise" as to how much they charge, and it isn't like there aren't a dozen alternatives including DIY. I'm the first to say that AWS is too expensive, and I vote with my wallet (and the company I work for by proxy). But I'll never claim that there's any gouging involved.

ballenf 6y ago
Price gouging is the practice of using outsized leverage in a particular market to charge excessive prices. Like snow shovels doubling in price after a snow storm. Or $10 water bottles after a hurricane. So for AWS the term is arguably correctly applied. But I'd be more worried about the market if AWS was artificially undercutting pricing because it would kill the incentive to create competitors or innovation in the space.
- sofixa 6y ago
  > Price gouging is the practice of using outsized leverage in a particular market to charge excessive prices. Like snow shovels doubling in price after a snow storm. Or $10 water bottles after a hurricane. > So for AWS the term is arguably correctly applied What outsized leverage have AWS had for a decade? There are multiple competitors at different levels, AWS are just better in terms of coverage/redundancy and amount of services.
  - pas 6y ago
    They are meaningfully better only in one thing: size = PR = network effects.
- jen20 6y ago
  > So for AWS the term is arguably correctly applied. Can you elicit the argument by which this is correctly applied?
  - ballenf 6y ago
    Point was just that having the "price on the tin" is irrelevant to whether there's price gouging going on. But, I guess the argument could be similar to the case against Apple for its iOS App Store: there are lots of competitors, but the lock-in arguably creates a market definition of Apple customers. AWS customers are largely locked-in and at the mercy of AWS prices. That's the argument. I'm not sure I buy it, but it's one perspective.

busterarm 6y ago

As a longtime ElasticSearch cluster admin/developer and Elastic Cloud customer, I don't feel bad for Elastic in the slightest and I'm psyched about this fork. The way they operate their cloud service leaves a lot to be desired and encourages maximum spend if you end up wanting to use it for anything demanding in production.

cassianoleal 6y ago
Not to mention the support is pretty slow to respond and rarely respond adequately. At my current client we started with their cloud and in a few months deployed our own in Kubernetes using the official operator. It's a lot cheaper and gives us fewer headaches this way.
- busterarm 6y ago
  My favorite moment as a customer dealing with all of the random times _they_ would take us down (even with triple redundancy) was their suggestion of operating a duplicate 3 datacenter cluster in production as a hot spare. They must think their customers are cash machines.
  - bombcar 6y ago
    I mean if they were truely best of breed they wouldn’t have needed a license change now would they?
    - busterarm 6y ago
      Indeed, although I'd say for a majority of situations, Amazon's hosted Elasticsearch service is a complete non-starter.
      - damon_c 6y ago
        Why is it a non-starter? I have a few ElasticSearch clusters that have been running for years with zero effort/downtime/hassle.

fastball 6y ago

I think the claim that Amazon is winning through "vendor lock-in" is pretty silly. Honestly anyone who can't quickly migrate the stuff they're hosting on AWS onto one of the many other cloud platforms is pretty bad at DevOps. If you're using K8S/Docker/etc it should be trivial. But even if you're not, the vast majority of AWS offerings were either built to be API-compatible with other existing tools (e.g. postgres-compat Aurora RDS), are literally identical to other services you can self-host (e.g. ElasticSearch) or others have built services compatible with AWS services (e.g. DigitalOcean's "Spaces" aim to be API identical to AWS S3 – you can literally use Amazon's S3 client libs to interact with various S3-compat services from other clouds). It's not "lock-in", it's providing a great all-in-one solution. You can host everything you want to host on AWS, which has good stability, good latency, etc. People are locked in because the DX around using AWS for everything your platform needs is just better than other platforms / having different services on different cloud providers (at least for many people).

real_joschi 6y ago
If you're not using any of the AWS services, that might be true but then you're also leaving a lot of potential on the table. If you're "cloud-agnostic" and could migrate away from AWS in the blink of an eye then you're paying for an overpriced VM offering and should probably migrate to a cheaper hosting provider immediately.
- busterarm 6y ago
  > If you're "cloud-agnostic" and could migrate away from AWS in the blink of an eye then you're paying for an overpriced VM offering and should probably migrate to a cheaper hosting provider immediately. No this is what everyone suggesting this does not get. The offerings are not equivalent. There are a baseline of services that the cloud providers offer that can be made functionally-equivalent. It's not just EC2 but more like EC2,S3,Lambda,RDS,DynamoDB,ECS,EKS (plus some others and of course the other cloud's equivalents). The secret sauce is in the APIs and permisioning and all of these available within the same VPC (talking to each other without paying bandwidth costs). "Cloud-agnostic" has _never_ meant "just VMs". Some of these services are majorly hard to duplicate on your own VMs as well. Feel free to implement "cheap VM hosting + S3" and burn cash on transit costs. Cheaper hosting providers do not give you this by miles.

minhazm 6y ago

Is it really price gouging for bandwidth? Or is bandwidth just really expensive in general? I honestly don't know. I would assume if it was actually much cheaper one of the cloud's would undercut the other to get customers.

GordonS 6y ago
It really is price gouging - bandwidth is actually cheap. A couple of comparisons: Oracle Cloud give you 10TB of bandwidth for free, with overage charged at around €7.5/TB. You can rent a VPS from the likes of Hetzner, and they will throw in 2-20TB of bandwidth for free, with overage charged at something like €1/TB - AWS charge an eye-watering €125 for each TB! I think the reason the big 3 (AWS, Azure, GCP) still charge such huge amounts is that they profit so greatly from it, and there is more than enough business to go round.
- martinald 6y ago
  I totally agree, but was stunned to see DigitalOcean charging $0.10/GB for outbound transfer for their new (quite cool imo) apps service. You do get 40GB-100GB included, but it means it's unusable for bandwidth heavy apps. They include 1TB (which is pooled across all VMs) and 0.01/GB on their standard platform.
  - GordonS 6y ago
    Yeah, I think Digital Ocean are really taking the piss there - proven by their existing $0.01/GB pricing, as you mentioned. I reckon DO have seen the big providers getting away with it, and they want a piece. Why wouldn't they, when there is apparently so little pushback?
    - martinald 6y ago
      I suppose you have object storage which is still $0.01/GB (plus includes 1TB to start with), so for most apps your bandwidth will come out of that for most files, so your 0.1/GB price is only for html/api/etc transfer. But still, it's annoying and seems completely arbitatary.

deleted 6y ago

[deleted]

notyourday 6y ago

As someone who tried buying services from ES and had to deal with their smug sales people that had a total disdain to those who wanted to give ES money, I am happy I will never need to deal with them in future.

prepend 6y ago

> In fact, Amazon contributes very little to open source in general, considering how much they take from it I don’t think this is a fact. Amazon seems to contribute pretty significantly according to the pages [0,1] they put out that describes their contributions. Not to mention their membership in OSS foundations like Linux Foundation. [2] You have the caveat about in relation to benefit they gain, but that’s pretty hard to measure. And I think isn’t really a good measure. I’d like to learn more about why you make such an absolute claim and maybe you have some better measure. I remember back in the 90s when big orgs (Microsoft, IBM) didn’t contribute to open source and can’t even think of any big orgs today that don’t contribute to open source. Even Oracle has big open source projects. [0] https://amzn.github.io/ https://amzn.github.io/ [1] https://aws.amazon.com/opensource/ https://aws.amazon.com/opensource/ [2] https://www.linuxfoundation.org/en/join/members/ https://www.linuxfoundation.org/en/join/members/

cbushko 6y ago
To me it looks like they mostly contribute to projects that are SDKs to use AWS.
aledalgrande 6y ago
Yeah and also what about projects like Firecracker? https://github.com/firecracker-microvm/firecracker/ https://github.com/firecracker-microvm/firecracker/
TechBro8615 6y ago
The absolute claim, aside from being at home in a rant on HN, comes from a cursory glance at https://github.com/amzn https://github.com/amzn, weighted by contributors and popularity, and compared to companies of similar size. Google, Microsoft and Facebook all build and maintain multiple open source projects that are hugely popular with people who use them outside of the company sandboxes. For example, people benefit from React without Facebook gaining much directly. (Facebook! If Facebook has any redeeming qualities, it's their open source contributions to the frontend ecosystem, although I promise you I could ascribe malicious intent to those as well...) Contrast that with Amazon. On their GitHub page, I see a few obscure projects amongst a bulbous array of AWS SDKs. To the sibling comment that asked about Firecracker -- I think Firecracker is awesome, and I did mention that in my original complaint. They even created it themselves! Well, a team of amazing engineers in Romania did. I have no personal insight into the matter, but it seems like they operate relatively independently from the AWS profit machine. Good for them too, it's incredible software. But I'm sure if they were to tell the story of how they got buy-in at Amazon to open source it, the same themes would come up -- how does Amazon benefit from this? In the case of Firecracker, the more people test it / harden it / run Doom on it, the more value Amazon can provide on its serverless platform. So again, unlikely to be purely altruistic intentions... but that's not to say there's anything wrong with that. I just find it all a bit distasteful in aggregate.

z77dj3kl 6y ago

My favorite "conspiracy theory" is that AWS intentionally creates stupidly verbose and numerous headers in all of their APIs just to up the bandwidth usage a few bytes per request at a time.

deleted 6y ago
[deleted]
cortesoft 6y ago
You don’t pay for the bandwidth of API calls to AWS

015a 6y ago

One thing which surprised me: Elastic has a market capitalization of ~$11B. I think that changes some of the more floaty ethical concerns. This is not a David vs Goliath situation. This is Goliath vs Super-Goliath. At this point, I'm much less interested in the drama of which mega-corp is screwing over the other. I'm more interested in: how does it affect me? When the titans are done trampling over the rest of us, which side benefits me the most? Its too early to tell, but it seems like it'll be Amazon. The product is more open. They have a demonstrated history of great support. Yeah, they gouge us on networking and everything else, but at least they're the devil we know, and buying into the OpenSearch ecosystem has a greater probability of being the more open solution into the next decade.

aardvarkr 6y ago
Uhmmm I’m pretty sure David vs Goliath is talking about scale between competition. Saying that $11B is Goliath just because you’re sitting at $1M doesn’t mean they’re not in a crazy mismatched fight against a $2T company. In the same way you could be in a David vs Goliath situation yourself if you with $1M in wealth tried to sue someone with $25K of wealth. Everything is relative. Doesn’t mean it’s not a crazy unfair mismatch that doesn’t deserve sympathy and regret.
- bombcar 6y ago
  When there are two large entities (and I take this to mean way larger than you or your company) then you’re better off rooting for the one that at least releases code you can use. If OpenSearch is truely open then in theory you can find another provider. But for ElasticSearch you’re stuck with them.
  - kfir 6y ago
    > But for ElasticSearch you’re stuck with them. I am a bit at a loss about that statement, you can run Elasticsearch on your on infrastructure
    - virgilp 6y ago

gsich 6y ago

No need to feel bad. The CEO sold a lot of his shares prior to the license change.

ampdepolymerase 6y ago

What's worse is the second order repercussions. Future open core/open source SaaSes will go straight to something like the Business Source License or the MongoDB license instead of traditional libre licenses. Amazon has done an incredible amount of damage to the open ecosystem.

randoramax 6y ago
Good. So the VCs will go back to playing like in the old days of shareware software licenses. It'll be good.

hintymad 6y ago

It sounds trivial to "wrap open source software", but surprisingly it is big value-add to thousands of companies. We can't just look at successful companies like Netflix to downplay the challenges of operating a service. Not every company knows how to operate complex systems under manageable cost. How many companies can really manage a Kafka cluster, let alone scaling it, for instance? Indeed, even companies that people deem powerful may screw up, if they don't get their culture or process right. Take Uber for example, for god damn five years, they still couldn't offer a service like EC2, let alone supporting persistent volumes. They still couldn't make their database provisioning on demand via an API. Their MySQL-based NoSQL solution was still based on FriendFeed's architecture and the APIs were hard to use. Yet they spent millions building a k8s replacement, building a GPU database, switching from mysql to postgres and back to mysql, etc and etc. So, yes, cloud companies like AWS buildd mere control planes to wrap open-source software, yet such seemingly mundane offering does bring values to many customers.

cle 6y ago
This is the most important point, IMO. Amazon's value add is not the software itself, it's the operation of the software. That includes a LOT of stuff, not just making sure it's running. It's security modeling and patching, compliance, DDOS protection, etc. Amazon's product is an army of ops engineers working 24/7 to keep your stuff secure and online. With that in mind, their behavior here makes a lot more sense, and comparing it with companies who have dramatically different products, like Facebook and Google, takes a lot of effort to understand the differences and what impact they have.
- harshaw 6y ago
  AWS just has engineers full stop. No specific ops engineers.
- stefan_ 6y ago
  You can rent a car but nobody would suggest the car manufacturer be paid nothing for the privilege just cause the rental company cracked car maintenance and how to fill the tank. The AWS "value add" is only value add in the context of being locked into AWS in the first place.

throwaway823882 6y ago

> Is Amazon too big and a bully? Is Starbucks too big and a bully? Sure, they force Mom and Pop shops to close by out-competing them, and that sucks. But bullying? I believe that's just Capitalism. Or say you're 6'6 and weigh 230LBS and you join a football team full of people who are 5'9 and 175lbs. Are you a bully just because you're bigger? ES basically handed them a platter with a goose laying golden eggs and a sign that read "Free Goose" and hoped they wouldn't try to make money off the eggs.

StavrosK 6y ago
> Is Starbucks too big and a bully? Sure, they force Mom and Pop shops to close by out-competing them, and that sucks. But bullying? I believe that's just Capitalism. I'd say it's more like a mom-and-pop coffee shop giving free coffee to patrons hoping to make money on cookies, and Starbucks coming in, taking the free coffee, opening a nice stand right next to the shop and selling the coffee they got for free. I'm having trouble justifying this ethically.
- throwaway823882 6y ago
  If they're giving you free coffee, and you sell the coffee you got for free... where's the harm? Starbucks is giving it away. Why would anyone buy your coffee rather than get it for free from the source? Now, assuming Starbucks charged you for the coffee, and you then re-sold it, this should also be fine. Starbucks is charging you presumably a rate with which they can recoup their costs. But if they are selling it below cost, they are clearly putting themselves at risk. A lot of businesses take this kind of risk as a strategic part of their business, like with making the coffee free. But you have to do it in a way that a competitor can't turn around to their advantage, or you're screwing yourself. Enter the concept of "not for resale". If a seller enters into a contract with a buyer, that contract can stipulate that the buyer can't resell the goods. Starbucks could theoretically require you to sign a contract saying you will not resell their coffee. That's pretty standard with licenses, even software licenses. ES must have known that their license did not forbid reselling. Yet they based their business model on this resellable coffee that they were giving away in order to make money on cookies. Is it Amazon's fault that ES chose a business model where they were selling coffee at a loss? Does Amazon have an ethical responsibility to keep ES's business afloat? Should we find any business unethical that tries to undercut the customer base of a rival, or take advantage of a rival's shaky business model? I think you have to come up with a whole framework for ethical competition, because one rule at a time isn't gonna capture it. (But I also think Capitalism is inherently unethical)

jrochkind1 6y ago

What ES wants of course is for Amazon to give them a cut of revenue from hosting ES. We already know Amazon isn't interested in doing that (either at all, or at whatever price ES wanted, we don't know that). They had no legal requirement to when ES was open source. So ES changed the licensing to no longer be open source. So, Amazon could... a) decide to give ES a cut after all, b) decide to stop hosting ES, or c) fork the last open source version. I don't think anyone is surprised they chose c? Presumably ES isn't either? Maybe ES thinks this will be good for them/bad for Amazon anyway, because they are hoping potential customers will abandon the Amazon fork and stay on the original ES fork? Not sure why they'd be confident in that exactly. Maybe they know what they're doing. As users/customers, we would rather have a choice of hosted vendors/platforms, and that it remain un-forked (so we can use/write software compatible with either vendor/platform). Competition is good for us as users/customers, that's in fact one of the reasons we choose open source, so no one vendor can set the hosting price all on their own without competition. We want to be able to choose among competitors for hosting, based on price, customer service, performance, uptime, whatever. But ES didn't want that, they didn't want hosting competition to exist -- at least not without permission and agreed upon cut for them -- because, I guess, hosting was how they planned to make money as a company to fund development as well as profits for investors etc. So they changed their license to no longer allow it. So of the possible outcomes remaining... this one seems as good as any for the user/customer, I guess? So, when you say "I'm doing my part by not building anything with vendor lock-in" -- I'm not sure which course you are suggesting. In fact, between ElasticSearch and new OpenSearch fork.. it's OpenSearch that is the one without vendor lock-in, right? OpenSearch is Apache licensed, and can be hosted by any vendor and still forked by anyone . It's ElasticSearch that has a license limiting what vendors can host it (without permission of ES), it's the one with vendor lock-in, right? So not building anything with vendor lock-in means... ?

galaxyLogic 6y ago
Good interesting points. Now Amazon will be the good guy because they will run open-source version, whereas ElasticSearch is not, if I understand you correctly. No single capitalist wants competitive markets. They want monopoly, for themselves. It is only when they don't have the monopoly or an easy way to get it that they cry for competitive markets. And that is good of course.

tssva 6y ago

"without violating the license a small startup put in place to stop them?" Elasticsearch was first released over a decade ago. ElasticSearch, now just Elastic, the company was founded over 9 years ago and now is public. Are they still a "small startup"? If so when does a company graduate from that status?

manigandham 6y ago

People want to pay for services, not software and licenses. They want turn-key solutions that are available via API and GUI, instantly and on-demand. This is the fundamental reason why AWS is so successful and the demand is constantly proven with every new product launch. Elastic (and other vendors) complaining about this instead of using it for their own success is a problem of their own making. At least a few companies are finally learning.

reflexe 6y ago

It it quite funny, elasticsearch is also kind of wrapping an open source library (Apache Lucene) and selling it as their own product.

jrochkind1 6y ago
Very good point. Could Elastic's business model survive if lucene adopted the SSPL license that Elastic has, saying it's the "spirit" of open source? That is an interesting question.
- pas 6y ago
  The problem is, that AWS is not really sharing the internals of their fork. Big corps always say that there's no point in open sourcing their fork, because it depends so much on internal systems. Which might be true, but in case of AWS and infrastructure the possibility of an AWS competitor similarly implementing those internal services would probably help the market... MySQL for example took a proprietary API (the mSQL API) and implemented an open software for it. Then Oracle bought MySQL. MariaDB started and it quickly dropped full compatibility. Options for the developers/users. Great. AWS/GCP/Azure on the other hand distorts the market. Just as Google/Apple/MS fund Chrome/Safari/Edge development from other sources, thereby distorting the market, forcing browsers to be free as in beer. (Which arguably hurts the market.)
  - jrochkind1 6y ago
    The announcement says "All of the software in the OpenSearch project is released under the Apache License, Version 2.0 (ALv2)." You are suggesting this is not correct?

scarface74 6y ago

There is absolutely nothing “cloud agnostic” about using Terraform. Every provisioner is specific to a cloud provider. If you are at any scale, moving a k8s cluster is the least of your issues.

dieters 6y ago

There are also ways out of vendor lock-in. Alternator comes to mind as a way of migrating DynamoDB workloads out to other cloud vendors or your own servers: https://docs.scylladb.com/using-scylla/alternator/ https://docs.scylladb.com/using-scylla/alternator/

dariusj18 6y ago

> Amazon is not doing it for any altruistic reason The beauty of OSS is that motives don't matter. If Amazon contributes and it's not detrimental in someway to the code, then it's a plus for anyone else who wants to use it.

DetroitThrow 6y ago
Precisely! While their business itself may need to be broken up, a community governed OSS project isn't bad for OSS when the alternative is a proprietary license that gives a single corp the ability to not contribute back or be exposed to virality. All this being said, progressive corporate taxes seem more enticing year after year.

raiflip 6y ago

It seems reductionist to say Amazon primarily wraps around open source. What about EC2? S3? Glue? DynamoDB? Many of the services that provide the most value are services Amazon has built out.

sofixa 6y ago
Om top of these, many of the core services that AWS themselves rely on, like SQS, SNS, Kinesis, Lambda, Cloudfront, ECS, Fargate, Elastic Beanstalk are mostly homemade
Nemo_bis 6y ago
EC2 uses the KVM hypervisor.

amzn-throw 6y ago

> they can invest more resources into developing it than the original team ever could. I know this is a popular narrative, but as someone who works on AWS, I think you would be shocked by how small the individual dev teams are that build and maintain the services that everyone uses. I'm not going to downplay the network effects involved. Of course AWS has a tremendous advantage in being able to standardize the customer billing, IAM, and EC2 Usage. And there are economies of scale. But individual AWS service teams are: * incredibly lean and focused * still have to make a profit on their own terms based on the infrastructure they build and the fees they charge customers * laser customer obsessed to solve people's (developer's) direct needs. I understand the community's concern about AWS investment and approach to OSS. But I can assure you (though you have no reason to believe me) that the goal is never to embrace, extend, then extinguish. It's all in the service of going where the customers are, and solving problems that they tell us they have. The profits are a byproduct. The "working backwards" process is no joke. We spend a lot of time figuring out what is the right thing for customers to build, start building it, and THEN we think about how do we make money from it.

waynesonfire 6y ago
they have as many people dedicated to the capability as needed. at least the capabilities have owners, which from my experience is not trivial for an organization to achieve. curious if others have noticed this as well (capabilities without clear owners)? what does this depend on? time? company size? both?
g9yuayon 6y ago
In the world of OSS, code speaks. A small team may not help the cause of Open Search. This article nails how code influences leadership in an OSS project: http://hypercritical.co/2013/04/12/code-hard-or-go-home http://hypercritical.co/2013/04/12/code-hard-or-go-home
systemvoltage 6y ago
Could you please shed some light on how many people would be behind a product like AWS Lambda or AWS CloudWatch? As an outsider, I would guess huge swaths of developers with a massive hierarchy. Buildings full of folks working on AWS services. I have no idea and extremely curious.

grumple 6y ago

Here are the main reasons to make open source software: To provide something for free, public use To get the world to help you maintain your software A reason to not make open source software: You want the exclusive right to offer that software as part of a paid service This is not the first instance of a company not understanding the last point there, and it won't be the last.

downrightmike 6y ago

This is Amazon's playbook. Make a direct competitor and squeeze the originals out. They did that with jewelry early on and then anyone they couldn't buy out they would under cut until they capitulated like diapers.com The Everything Book by Brad Stone goes over this in detail. Clearly anti-competitive monopolistic actions are taken constantly by Amazon. The only reason they aren't trust busted is because the common line of reasoning is that consumers pay less for goods, but this is being looked at because shouldn't competition be lowering prices. IE if Amazon hadn't killed diapers.com, wouldn't diapers be cheaper overall? And the answer is they should be, but the government hasn't caught up. Once they start getting into the weeds, they'll see example after example of monopoly behavior destroying competitors and ultimately raising prices on consumers.

marvindanig 6y ago

Coincidentally, AWS hasn't open-sourced anything that they use internally. Zilch. Nada. And yet they are using 'open source' developed by another firm (smaller is inconsequential here) to market themselves. IMO, FOSS licensing is completely broken. Its definitions (of what is free/open) are from a boomer's era that is no longer sustainable. At least I wouldn’t want any of my FOSS projects become “corporate strategy” of any particular proto-nation.

pas 6y ago
That made me wonder which megacorp did? Google published a lot of papers about their architecture, but no code. They have many open source projects modeled after internal tools (eg. bazel), but there's no search/GFS/mapreduce/monorepo OSS project from them. MS has VSCode, that's great, and they even open sourced .NET. But Azure is a full black box, just like (almost) every Windows component. (Finally calc.exe is OSS!)
PHGamer 6y ago
gplv3 was supposed to fix this though no?
zokier 6y ago
> Coincidentally, AWS hasn't open-sourced anything that they use internally. Zilch. Nada. Firecracker. s2n. Not much, I agree

api 6y ago

You described the cloud lock in model very well, especially the bandwidth part. What they charge for outbound is nuts, and the other clouds are not much better. Inbound is free of course to make it easy to send your data in but costly to get it out.

brutal_chaos_ 6y ago

Did the BSDs do it wrong, too? Apple uses a lot of FreeBSD software that they turn around and sell for profit. How about PostgreSQL, didn't Amazon fork that as well? My point is, there's nothing wrong with forks nor companies forking if the license allows for it. It's up to the developers to choose an appropriate license to not be forked/ripped off, if they so desire. I personally am against modern day corporate America, but I can't blame them for this. The software is given away free/libre/gratis to be forked by whomever. Perhaps to combat this, one should choose a non "Open Source(TM)" license, but a source availbe license. E.g https://mariadb.com/bsl11/ https://mariadb.com/bsl11/ (not my personal favorite, just an example). Also, I definitely agree with/do the same: > Personally I'm doing my part by not building anything with vendor lock-in. It's great to be able to deploy to any cloud, if you value either robustness or flexibility.

JoshTriplett 6y ago

When a product that was previously Open Source changes to a non-open license, it's not uncommon for someone to pick up the last Open Source version and fork it, and release that for others to use and collaborate on. That's always going to be a good thing; that means people who care about Open Source licensing continue to have a version to use and collaborate on.

jakear 6y ago

The main thing I look at in this situation is their approaches to security. ES decided authentication is a paid-only feature implemented via closed source proprietary code, and the result has been countless PII leaks; now sure you could say that’s the developers’ fault for making the endpoints internet-accessible, but when the system has been designed from the start to both be insecure and hold PII, you have to place some blame on the provider as well. Amazon on the other hand developed a free and open source auth plugin and anyone is able to deploy it no problem.

killingtime74 6y ago

Only good one I can think of is Amazon Java, not sure how essential it is though

bitonp 6y ago

I am with you. The release could read: "having subsumed the opensource version of ES, we are now relicensing, calling it our own, and would be really happy if the opensource community would lie to contribute, because actually we don't totally understand how this product works. Many thanks to all who help us" The issue with AWS version are many fold, but the main one is that it forces extra usage of expensive EC2 units, for the following reasons: 1. Blue/Green updates -> Start a new cluster with new version, lock current cluster, copy over all the data (can take over a day), at the same time write to both clusters, when finished, lock both clusters while endpoints are swapped over, unlock new cluster, trash old cluster. During this process Customer pays for both. - Solution, done properly, fire up new node with new version, swap it in, wait for 'green', take out old... wait for 'green'.. rinse and repeat. Result.. system never goes down, endpoints remain the same, less cost to customer. 2.There are a minimum of the folloowing node types: - Master (small) x3 - Data (heavy on storage, medium on memory) - Hot Data (very heavy on memory as shards have to be held in memory) - Coordinating (query) nodes, heavy on memory, light on storage (cos there is no real storage) - Ingest (same as Query). - voting only.. tiny - AI/ML heavy on memory and storage.. cos they do real work In the AS world they have: - Masters - Data - Hot Data (cos they are really pricey) The Data nodes do all teh functionality of the data, ingest and query nodes. Th emain query always gores to a data node, while it passes to other data nodes to get the data out, then aggregates locally to it... so its incorrect usage of the system. A data node should only ever deal with its own data, and pass the results it finds back to a node away from the data. As your system is squeezed.. you add data nodes... not coordinating/Query or Ingest nodes, which you probably need. But thats more money into the coffers. 3. Their userhasging function is also old (2011 vesrion of bcrypt) , and fails. Any static password produces different results everytime you use it.. at least on the current opendistro. So you are forced into either not using security, or using proper security, which can be cumbersome across the cluster (if rolling your own). however.. on the cloud version they have base level security working, so thats Ok.. they arent using their own 'open' software. I could go on.. but its all flaky, and misunderstood at the core developer level. Toput on record, I have spoken twice to teh core development team to describe how updates should happen.. the last time 18 months ago (by 'spoken' I mean a face to face video call) .. so they know hoe it should be done.. but dont do it. 4. I have noticed that there are some functions/methods available on the setup in Amazon Linux, that dont exist on other Linux versions (centOs/Debian) that are security related. AWS Linux is 'lifted' from Redhat.. so another piece of software that they didnt write.. but obviously Redhat are happy with this. Maybe they got the licensing deal sorted .. who knows. basically gores like this: "Who would actually install, in production, what essentially is very close to pirated software?" Roll your own.. its a bit of grunt work up front, then 50% the cost of the cloud version.

llcoolv 6y ago

So VSCode, which is basically slomo Sublime Text for retards and Typescript which is usable only for prototyping are a good enough compensation for having most of their income from Linux (on Win and Azure) after all the almost criminal FUD they poured on Linux[0]? Come on, get real. 0. https://en.wikipedia.org/wiki/Halloween_documents https://en.wikipedia.org/wiki/Halloween_documents