18 ms·

Amazon Elasticsearch Service Is Now Amazon OpenSearch Service

81 points by daigoba66 5y ago

google234123 5y ago
This is consequence of open source.
- 0x500x79 5y ago
  Are you claiming this is a bad thing? Your statement doesn't have a whole lot of meat in it without elaborating a bit. There were a lot of decisions and changes that were made that led to this happening, and we have no idea how it ends up. I personally am interested to see what happens in the elasticsearch versus opensearch space. I want to see how this experiment plays out.
- josephcsible 5y ago
  This is the consequence of closed source. If Elasticsearch were still open source, then Amazon wouldn't have had any reason to do this. Elastic is getting exactly what they deserve.
  - Sebb767 5y ago
    If they would have been closed source from the get-go, Amazon would have had a far harder time cloning their product or even creating a service based on it to begin with. So yes, this is a consequence of them starting out open source.
    - josephcsible 5y ago
      If they would have been closed source from the get-go, they'd have even fewer customers than they do today with Amazon "stealing" all of them.
      - Sebb767 5y ago
        Possibly, I didn't refute that (and I actually agree). However, the point that this fork would likely not have happened without them being open source is completely valid.
  - echelon 5y ago
    Open source didn't go far enough in this case. It should have mandated the platform itself - from the billing pieces, to the hardware imaging, user management, and more - was all entirely open source.
    - josephcsible 5y ago
      Open source can't do that, literally. Any license that tries to impose "copyleft" on the whole platform like that will necessarily fail to meet the Open Source Definition.
      - echelon 5y ago
        You're arguing semantics, not intent. Just call it something new if definitions matter so much. "Fair Play Source", "Fair and Open", etc. Fuck open source the way Amazon likes it. Look at how the giants abuse it to crush smaller innovators. It's not rewarding the original authors. It's turning them into labor slaves whose collective work is captured and monetized, with little hope of fair competition or reward. Might as well call it "open slavery" and imagine Amazon as some Jabba the Hut creature. The work was intended to be open. They just didn't imagine the giant Jabba slug eating the whole pie and freezing them out. Unintended, yet fatal, consequences. The DOJ should look very closely at this behavior. It's anticompetitive af and wholly against the spirit and intent of the original authors.
        Sebb767 5y ago
        Amazon is working on a fully open version that you can fork, edit and host yourself. Elastic closed their product and is changing their client libraries to not support the open version. In fact, this whole thing started because Elastic got envious of Amazons revenue, which is not quite the open source spirit. I mean, I get why Elastic is not happy. But in this case, neither intent is pure, but at least the result on the Amazon side is.
        josephcsible 5y ago
        > But in this case, neither intent is pure, but at least the result on the Amazon side is. Indeed. Amazon may be doing the right thing for the wrong reasons, but that's still better than doing the wrong thing for the wrong reasons, which is what Elastic did.
simlevesque 5y ago
I changed from AWS managed ElasticSearch to Elastic.co a couple of months ago. Should have done it sooner.
- atonse 5y ago
  Why? Tech or on principle? We are using Amazon ES but leaves a lot to be desired. But I don’t know how much of that is just ES and how much is Amazon
  - DanielVZ 5y ago
    We are doing the transition and everything is just so smooth. Adding new nodes is easy and fast. There's an actually good and consistent UI. Last time we tried to upgrade our cluster version on AWS we ended up waiting for weeks until it randomly updated mid day on a busy day and broke everything. It's a ton of small things that really make a difference.
    - longcommonname 5y ago
      How do you manage user access to the management console?
    - EschatonCometh 5y ago
      You can upgrade instantly on AWS, it doesn't take weeks. A lot of problems with AWS Elasticsearch Service are caused by corporations trying to squeeze a dollar out of a dime. T2 and T3s with no Master Nodes, let alone three, leads to a lot of problems. When everything lives in MMAP'd memory and you are nickel and dime'n it with 4GB and one CPU. You're going to have a bad day.
      - DanielVZ 5y ago
        Yeah that was for sure not the case for 3 m5.large master nodes and several r4.2xlarge nodes. The update was instant, what wasn't instant was the wait for the update. IDK if it was the maintenance window (which was set up for a weekend) or some bug, but we DID end up waiting weeks until it randomly updated.
RONROC 5y ago
Here’s what Elasticsearch thinks: https://news.slashdot.org/story/21/08/07/0040229/elasticsearch-keeps-fighting-open-source-fork-by-amazon-aws https://news.slashdot.org/story/21/08/07/0040229/elasticsear...
Sebb767 5y ago
As far as I'm aware, this is the first major "hostile" fork AWS has done. In fact, it's probably one of the largest ones overall. It will be really interesting to see how this plays out in the long run.
- echelon 5y ago
  It's good fodder for the antitrust suit.
- stupendousyappi 5y ago
  Google certainly didn't support Amazon forking Android into FireOS. I believe that was just about when Google began migrating tons of APIs from the Android OS to Google Play Services.
josephcsible 5y ago
Remember that in this story, as strange as this is to say, Amazon is the good guy. Elastic changed their products from a FOSS license to a proprietary one, and Amazon did exactly what you're supposed to do in that case: fork and continue development of the last FOSS version.
- wmf 5y ago
  They changed the license after Amazon took all their revenue.
  - josephcsible 5y ago
    I don't fault Elastic for choosing to change their license. I fault them for their choice of the new one. They should have moved to the AGPL instead, like Grafana did under basically the exact same circumstances.
    - whateveracct 5y ago
      yep - not going AGPL is just greed
  - zokier 5y ago
    > We have experienced significant growth, with revenue increasing to $608.5 million in the year ended April 30, 2021 from $427.6 million in the year ended April 30, 2020 and $271.7 million in the year ended April 30, 2019, representing year-over-year growth of 42% for the year ended April 30, 2021 and 57% for the year ended April 30, 2020 https://s2.q4cdn.com/265747582/files/doc_downloads/2021/Elastic-Annual-Report-Fiscal-Year-2021.pdf https://s2.q4cdn.com/265747582/files/doc_downloads/2021/Elas... Revenue for FY2018 and 2017 was $160M and $88M respectively. Is that what Amazon taking all their revenue looks like?
    - true_religion 5y ago
      The fact that they grew despite Amazon doesn't mean that AWS Elasticsearch wasn't a viable competitor. It just means that at the moment it was not an existential threat. However, when competing with a literally A in FAANG, one must be very careful about writing them off. To compare, I knew a company that 10 years ago made really great business intelligence integrations into Excel. They'd operated for 15+ years and were a major local employer. They thought Palantir was going to be their competitor, but then Microsoft came out with Power BI. At first, revenue grew because Microsoft's marketing shined a great spotlight on the field (plus MS had only entered because the field was growing in the first place). Then as the situation matured, MS's capacity to offer integrations to the rest of the stack became a key sales point. Like AWS, the point of investing in their stack isn't one particular product but the fact that all the billing, support, and integrations happen via one company. You can even accept it if any particular product is deficient (e.g., some say that about Teams or Skype), but in the case of an open-source product like Elastic--the value of the software itself is 100% the same. AWS was going to eventually kill Elastic. Not just because of all I wrote above, but also because Elastic's got a significant part of its revenue by selling Elastic as a distribution on top of AWS. Anyone would want to cut out the middleman and simply use AWS's version even if it lagged behind the current Elastic.
fshbbdssbbgdd 5y ago
If I was a mole for Amazon working at Elasticsearch, I can’t think of a better way to accelerate the shift onto AWS’s offering than breaking client comparability with previous versions of Elasticsearch. If a customer needs to upgrade the client for some reason, they are forced to update the server as well. Maybe they’ll just update the client and server to OpenSearch while they’re at it so they don’t have any more forced updates in the future.
- kjsingh 5y ago
  like that didn't happen
thayne 5y ago
Good. I wonder if some of this mess could have been avoided if AWS had used a different name to begin with.
dhd415 5y ago
While this is tangentially related to Amazon's fork of Elasticsearch, it's more related to Elastic's trademark infringement suit against AWS's use of the trademarked Elasticsearch name in their managed service, especially egregious given AWS's attempt to portray it as a "partnership" with Elastic in a since-deleted tweet: https://mobile.twitter.com/ppgosavi/status/1179181900969037826 https://mobile.twitter.com/ppgosavi/status/11791819009690378...
- mohanmcgeek 5y ago
  I doubt Amazon cares about that suit. If anything, it's elastic that hugely benefited from calling their product after countless Amazon products that have the word elastic in them. Amazon can show that EC2 predates the company by half a decade atleast
thehappypm 5y ago
Morale of the story: licenses matter. Get them right.
jrochkind1 5y ago
> One of the most important reasons is the freedom to use that software where and how they want. True for me, I agree.
dumpsterdiver 5y ago
In OSS, what a company giveth, a company cannot taketh away.
api 5y ago
Use a copyleft license. If you're not, you are just giving free labor to billion dollar companies.
- mohanmcgeek 5y ago
  In this case, which copyleft license would help them? If they use GPL, it makes no difference as Amazon can still run it but if they use AGPL orgs would blacklist this because they don't want to be forced to open-source the rest of their code for their "elasticsearch integration"
  - josephcsible 5y ago
    AGPL worked fine for Grafana for this purpose.
    - mohanmcgeek 5y ago
      Too early to tell. People might still be running the older non agpl version
binarymax 5y ago
So many people in this discussion take sides. The only thing I see, is divergence. APIs will now evolve separately, and compatibility will break soon. I think this competition and diversification will be good for innovation in search. But it’s also going to be confusing for a while. If you use Elasticsearch in Amazon, you better decide very quickly whether you want to hitch your wagon to the OpenSearch future. Migrating is going to be a pain in a year or two.
- altdataseller 5y ago
  Why would most companies need to migrate if they're happy with the features in a past version? Why not stick with it (assuming there are no major security flaws?)
  - binarymax 5y ago
    Well, how far is Amazon going to take OpenSearch? Elastic is all in on innovating and building their flagship product forever with significant dedication. AWS just needs something good enough as an offering to go with the rest of their OK cloud services.
- Sebb767 5y ago
  I think most people can see both sides of this conflict. But, as you yourself pointed out at the end of your comment, finding a side to pick quickly is important.
- theamk 5y ago
  This didn't happen with MySQL/MariaDB split -- they are still mostly compatible. Granted, it looks like Elastic is trying to break compatibility pretty hard (see the new version checks in clients for example) but hopefully Amazon will release universal connectors, it certainly in the Amazon's interests.
  - binarymax 5y ago
    Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense vector search (it has before been only sparse vector search for Lucene based platforms). Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuecomment-910719424 https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.html https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.
ckdarby 5y ago
RIP Elastic

81 points by daigoba66 5y ago

google234123 5y ago

This is consequence of open source.

0x500x79 5y ago
Are you claiming this is a bad thing? Your statement doesn't have a whole lot of meat in it without elaborating a bit. There were a lot of decisions and changes that were made that led to this happening, and we have no idea how it ends up. I personally am interested to see what happens in the elasticsearch versus opensearch space. I want to see how this experiment plays out.
josephcsible 5y ago
This is the consequence of closed source. If Elasticsearch were still open source, then Amazon wouldn't have had any reason to do this. Elastic is getting exactly what they deserve.
- Sebb767 5y ago
  If they would have been closed source from the get-go, Amazon would have had a far harder time cloning their product or even creating a service based on it to begin with. So yes, this is a consequence of them starting out open source.
  - josephcsible 5y ago
    If they would have been closed source from the get-go, they'd have even fewer customers than they do today with Amazon "stealing" all of them.
    - Sebb767 5y ago
      Possibly, I didn't refute that (and I actually agree). However, the point that this fork would likely not have happened without them being open source is completely valid.
- echelon 5y ago
  Open source didn't go far enough in this case. It should have mandated the platform itself - from the billing pieces, to the hardware imaging, user management, and more - was all entirely open source.
  - josephcsible 5y ago
    Open source can't do that, literally. Any license that tries to impose "copyleft" on the whole platform like that will necessarily fail to meet the Open Source Definition.
    - echelon 5y ago
      You're arguing semantics, not intent. Just call it something new if definitions matter so much. "Fair Play Source", "Fair and Open", etc. Fuck open source the way Amazon likes it. Look at how the giants abuse it to crush smaller innovators. It's not rewarding the original authors. It's turning them into labor slaves whose collective work is captured and monetized, with little hope of fair competition or reward. Might as well call it "open slavery" and imagine Amazon as some Jabba the Hut creature. The work was intended to be open. They just didn't imagine the giant Jabba slug eating the whole pie and freezing them out. Unintended, yet fatal, consequences. The DOJ should look very closely at this behavior. It's anticompetitive af and wholly against the spirit and intent of the original authors.
      - Sebb767 5y ago
        Amazon is working on a fully open version that you can fork, edit and host yourself. Elastic closed their product and is changing their client libraries to not support the open version. In fact, this whole thing started because Elastic got envious of Amazons revenue, which is not quite the open source spirit. I mean, I get why Elastic is not happy. But in this case, neither intent is pure, but at least the result on the Amazon side is.
        josephcsible 5y ago
        > But in this case, neither intent is pure, but at least the result on the Amazon side is. Indeed. Amazon may be doing the right thing for the wrong reasons, but that's still better than doing the wrong thing for the wrong reasons, which is what Elastic did.

simlevesque 5y ago

I changed from AWS managed ElasticSearch to Elastic.co a couple of months ago. Should have done it sooner.

atonse 5y ago
Why? Tech or on principle? We are using Amazon ES but leaves a lot to be desired. But I don’t know how much of that is just ES and how much is Amazon
- DanielVZ 5y ago
  We are doing the transition and everything is just so smooth. Adding new nodes is easy and fast. There's an actually good and consistent UI. Last time we tried to upgrade our cluster version on AWS we ended up waiting for weeks until it randomly updated mid day on a busy day and broke everything. It's a ton of small things that really make a difference.
  - longcommonname 5y ago
    How do you manage user access to the management console?
  - EschatonCometh 5y ago
    You can upgrade instantly on AWS, it doesn't take weeks. A lot of problems with AWS Elasticsearch Service are caused by corporations trying to squeeze a dollar out of a dime. T2 and T3s with no Master Nodes, let alone three, leads to a lot of problems. When everything lives in MMAP'd memory and you are nickel and dime'n it with 4GB and one CPU. You're going to have a bad day.
    - DanielVZ 5y ago
      Yeah that was for sure not the case for 3 m5.large master nodes and several r4.2xlarge nodes. The update was instant, what wasn't instant was the wait for the update. IDK if it was the maintenance window (which was set up for a weekend) or some bug, but we DID end up waiting weeks until it randomly updated.

RONROC 5y ago

Here’s what Elasticsearch thinks: https://news.slashdot.org/story/21/08/07/0040229/elasticsearch-keeps-fighting-open-source-fork-by-amazon-aws https://news.slashdot.org/story/21/08/07/0040229/elasticsear...

Sebb767 5y ago

As far as I'm aware, this is the first major "hostile" fork AWS has done. In fact, it's probably one of the largest ones overall. It will be really interesting to see how this plays out in the long run.

echelon 5y ago
It's good fodder for the antitrust suit.
stupendousyappi 5y ago
Google certainly didn't support Amazon forking Android into FireOS. I believe that was just about when Google began migrating tons of APIs from the Android OS to Google Play Services.

josephcsible 5y ago

Remember that in this story, as strange as this is to say, Amazon is the good guy. Elastic changed their products from a FOSS license to a proprietary one, and Amazon did exactly what you're supposed to do in that case: fork and continue development of the last FOSS version.

wmf 5y ago
They changed the license after Amazon took all their revenue.
- josephcsible 5y ago
  I don't fault Elastic for choosing to change their license. I fault them for their choice of the new one. They should have moved to the AGPL instead, like Grafana did under basically the exact same circumstances.
  - whateveracct 5y ago
    yep - not going AGPL is just greed
- zokier 5y ago
  > We have experienced significant growth, with revenue increasing to $608.5 million in the year ended April 30, 2021 from $427.6 million in the year ended April 30, 2020 and $271.7 million in the year ended April 30, 2019, representing year-over-year growth of 42% for the year ended April 30, 2021 and 57% for the year ended April 30, 2020 https://s2.q4cdn.com/265747582/files/doc_downloads/2021/Elastic-Annual-Report-Fiscal-Year-2021.pdf https://s2.q4cdn.com/265747582/files/doc_downloads/2021/Elas... Revenue for FY2018 and 2017 was $160M and $88M respectively. Is that what Amazon taking all their revenue looks like?
  - true_religion 5y ago
    The fact that they grew despite Amazon doesn't mean that AWS Elasticsearch wasn't a viable competitor. It just means that at the moment it was not an existential threat. However, when competing with a literally A in FAANG, one must be very careful about writing them off. To compare, I knew a company that 10 years ago made really great business intelligence integrations into Excel. They'd operated for 15+ years and were a major local employer. They thought Palantir was going to be their competitor, but then Microsoft came out with Power BI. At first, revenue grew because Microsoft's marketing shined a great spotlight on the field (plus MS had only entered because the field was growing in the first place). Then as the situation matured, MS's capacity to offer integrations to the rest of the stack became a key sales point. Like AWS, the point of investing in their stack isn't one particular product but the fact that all the billing, support, and integrations happen via one company. You can even accept it if any particular product is deficient (e.g., some say that about Teams or Skype), but in the case of an open-source product like Elastic--the value of the software itself is 100% the same. AWS was going to eventually kill Elastic. Not just because of all I wrote above, but also because Elastic's got a significant part of its revenue by selling Elastic as a distribution on top of AWS. Anyone would want to cut out the middleman and simply use AWS's version even if it lagged behind the current Elastic.

fshbbdssbbgdd 5y ago

If I was a mole for Amazon working at Elasticsearch, I can’t think of a better way to accelerate the shift onto AWS’s offering than breaking client comparability with previous versions of Elasticsearch. If a customer needs to upgrade the client for some reason, they are forced to update the server as well. Maybe they’ll just update the client and server to OpenSearch while they’re at it so they don’t have any more forced updates in the future.

kjsingh 5y ago
like that didn't happen

thayne 5y ago

Good. I wonder if some of this mess could have been avoided if AWS had used a different name to begin with.

dhd415 5y ago

While this is tangentially related to Amazon's fork of Elasticsearch, it's more related to Elastic's trademark infringement suit against AWS's use of the trademarked Elasticsearch name in their managed service, especially egregious given AWS's attempt to portray it as a "partnership" with Elastic in a since-deleted tweet: https://mobile.twitter.com/ppgosavi/status/1179181900969037826 https://mobile.twitter.com/ppgosavi/status/11791819009690378...

mohanmcgeek 5y ago
I doubt Amazon cares about that suit. If anything, it's elastic that hugely benefited from calling their product after countless Amazon products that have the word elastic in them. Amazon can show that EC2 predates the company by half a decade atleast

thehappypm 5y ago

Morale of the story: licenses matter. Get them right.

jrochkind1 5y ago

> One of the most important reasons is the freedom to use that software where and how they want. True for me, I agree.

dumpsterdiver 5y ago

In OSS, what a company giveth, a company cannot taketh away.

api 5y ago

Use a copyleft license. If you're not, you are just giving free labor to billion dollar companies.

mohanmcgeek 5y ago
In this case, which copyleft license would help them? If they use GPL, it makes no difference as Amazon can still run it but if they use AGPL orgs would blacklist this because they don't want to be forced to open-source the rest of their code for their "elasticsearch integration"
- josephcsible 5y ago
  AGPL worked fine for Grafana for this purpose.
  - mohanmcgeek 5y ago
    Too early to tell. People might still be running the older non agpl version

binarymax 5y ago

So many people in this discussion take sides. The only thing I see, is divergence. APIs will now evolve separately, and compatibility will break soon. I think this competition and diversification will be good for innovation in search. But it’s also going to be confusing for a while. If you use Elasticsearch in Amazon, you better decide very quickly whether you want to hitch your wagon to the OpenSearch future. Migrating is going to be a pain in a year or two.

altdataseller 5y ago
Why would most companies need to migrate if they're happy with the features in a past version? Why not stick with it (assuming there are no major security flaws?)
- binarymax 5y ago
  Well, how far is Amazon going to take OpenSearch? Elastic is all in on innovating and building their flagship product forever with significant dedication. AWS just needs something good enough as an offering to go with the rest of their OK cloud services.
Sebb767 5y ago
I think most people can see both sides of this conflict. But, as you yourself pointed out at the end of your comment, finding a side to pick quickly is important.
theamk 5y ago
This didn't happen with MySQL/MariaDB split -- they are still mostly compatible. Granted, it looks like Elastic is trying to break compatibility pretty hard (see the new version checks in clients for example) but hopefully Amazon will release universal connectors, it certainly in the Amazon's interests.
- binarymax 5y ago
  Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense vector search (it has before been only sparse vector search for Lucene based platforms). Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuecomment-910719424 https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.html https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.

ckdarby 5y ago

RIP Elastic