12 ms·

Opening the code of our X-Pack features

159 points by Benfromparis 9y ago

__emmerich 9y ago
Their licence is still proprietary, though. At the least people with a technical curiosity in X-Pack will be able to install it. Technically it's illegal, but Elastic probably doesn't care until the moment they start gaining momentum.
- busterarm 9y ago
  I wouldn't put it past the Elastic team to include some sort of phone-home mechanism here. Nor would I put it past them to consider legal demand letters as part of their sales funnel.
  - Redsquare 9y ago
    Based on what exactly?
    - busterarm 9y ago
      Based on a previous experience being in their sales funnel and speaking to their reps at re:Invent. They have an incredible product that I can't live without, it's just incredibly painful trying to actually pay them money for something. Honestly, I think their real play here is getting community contributions to fix stuff that's very broken and also to edge out Siren Solutions. SearchGuard, Kibi and Sentinl are all _very_ good tools.
      - tylerhannan 9y ago
        I'm sorry to hear it was painful to pay us...I wish that wasn't the case. If you want, I'd be happy to talk more about it so that we can understand where we can improve. Feel free to email me tyler.hannan<at>elastic.co <disclaimer: I work at Elastic in Developer Relations>
        arrayjumper 9y ago
        Not OP. One reason i find it is hard to pay you is because there is no way to buy your product without talking to someone. The pricing is not listed anywhere on your website.
        busterarm 9y ago
        That changed Jan 1 while were in the middle of figuring out what we were going to pay them -- then some X-Pack features disappeared from the "Gold" plan and the cost they quoted us was _way up_ from what we were evaluating. There actually still is a pricing page somewhere on the site, but you have to dig for it, and I don't think it's correct anymore.
  - tylerhannan 9y ago
    In 6.3, telemetry (‘phone home’) is an opt-in only feature for both the free and paid components of X-Pack. We won’t be tracking any identification, so even if users opt-in to telemetry, there is no way to turn that into a sales strategy. Suing users is a pretty terrible business practice and not, at all, a funnel creation opportunity. Re. License, there are a few FAQs on https://www.elastic.co/products/x-pack/open#faq https://www.elastic.co/products/x-pack/open#faq that may help clear up any misunderstandings. Free features are free and governed by the Elastic EULA but are enabled by default. Paid features are available through an opt-in trial and, if they help solve a problem, can be purchased. <disclaimer: I work at Elastic in Developer Relations>
    - ohstopitu 9y ago
      and this trial can't be disabled (to use them for an unlimited amount of time) by someone by modifying your source because your license prohibits that? Am I correct in that assumption?
- passionforcode 9y ago
  I think open sourcing the code give opportunities to others to learn from the code and create own alternatives. What drive the community to involve in their software
  - richdougherty 9y ago
    The main benefit may be that users can more easily use the code if they can read it. If you can see the code then you can understand how it works exactly, without relying on documentation to be accurate. You can also more easily debug your application if you have the source of third-party libraries. For writing clones or alternatives, it might be better not to read any proprietary code: https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design
    - passionforcode 9y ago
      You have a reason. That will be one of the main reasons. Sometimes I use open source projects to find a way to create something. Check many solutions can give an idea how to solve the puzzle. Make this projects open source can give me an idea to create plugins. Ofcourse still need think for yourself if it fits your needs, quality and if you not are struggle with possible copyrights.
- continuations 9y ago
  If their license is still proprietary what do they mean by "opening" the code?
  - ryukafalz 9y ago
    They mean you can see the code. Note that they never say they're "open sourcing" it - it's not open source by the Open Source Definition. This is the license the X-Pack code is under: https://www.elastic.co/eula https://www.elastic.co/eula
    - bluecanary37 9y ago
      15 mentions of "open source" in the blog post and 12 more in the "Opening X-Pack" page, ignoring the other mentions of "open".
Jedd 9y ago
Eleven uses of the word free, all used in the gratis, not libre, sense of the word. 'Open' is in the same category for me now as 'cloud' -- too nebulous, too little consensus on what it actually means, often used in a 'don't worry about the details' hand-wavy way.
iramiller 9y ago
The important part is near the bottom, free components of X-Pack will no longer require the cumbersome registration process. Also of some concern is that extra care will be required to find a version of the software that is Apache 2.0 licensed as the proprietary licensed X-Pack components will be included by default in the code base instead of as a separate install. --- "Also, X-Pack features will now be bundled into the default distribution. All free features are included and enabled by default and will never ‘expire’, and commercial features are opt-in via a trial license. The license for free features never expires,you no longer need to register to use these capabilities. In addition to this, an Apache 2.0-only distribution will be created for download."
btown 9y ago
This is incredibly user-hostile, and if you read this expecting this is an open-source release, consider yourself clickbaited. Want to contribute to the open-source version of Elasticsearch? Want to ensure you're running an open-source database without running a custom EULA past your legal department? Well, make sure that after 6.3 is released, you don't accidentally download the default distribution, or browse the official Github repository, or clone from Github, as all of those will have components subject to an as-yet-unreleased, non-OSI-approved EULA (see the final paragraph in https://www.elastic.co/products/x-pack/open https://www.elastic.co/products/x-pack/open), which may have clauses that trigger liability to Elastic if you accidentally flip a switch or look at the wrong thing. (I am not a lawyer, this is not legal advice, but I shouldn't have to be to know if I can use software freely, and I can only guess worst-case scenarios because the terms aren't released.) If Elastic were truly committed to transparency, they would release the terms of their new EULA immediately, provide a way to access the head of the open-source portion of the repository without accepting the EULA, and make it clear what the pricing ramifications are if one opts into certain flags or reads certain files in a commercial capacity. (Pricing requires a custom quote, but a quick Google search suggests that the lowest tier in which REST and node-to-node communications are encrypted begins in the tens of thousands of dollars - not the kind of thing you want to take lightly.) Now more than ever, it's important that projects like https://github.com/floragunncom/search-guard https://github.com/floragunncom/search-guard , a free and open-source (Apache 2.0) alternative to some of these X-Pack components, are supported, and that the word is spread around them. (EDIT: Yes, they have commercial components, but those are in a separate repository, so you avoid any entanglements. Why isn't Elastic doing the same?) And hopefully, if Elastic doesn't maintain a fully-OSS repo, someone will create and maintain a friendly fully-OSS fork that can be combined with third-party software to carry on the legacy of this otherwise excellent database in the right way.
- tylerhannan 9y ago
  One of the awesome things about our products is that people care deeply about them. There is one thing, in particular, I want to address. 'If Elastic were truly committed to transparency, they would release the terms of their new Eula immediately...' We are working on it. This change is not one that was taken lightly and we have to ensure we have the right language for when we do release the license, etc. We will put it online as soon as we are able. We are committed to transparency and, also, we are committed to our users. Yes, we are asking for a fair bit of trust...and I hope we continue to prove ourselves worthy of that trust. <disclaimer: I work at Elastic in Developer Relations>
jlg23 9y ago
Background: They introduced X-Pack by bundling it with the default distribution as a time-limited trial without explicitly stating that it is just a demo. People who did the update were bitten weeks later when it just stopped working because the demo license had expired. [Documentation of this was ridiculously bad and I only learned through this post that there apparently was some way to get a free license (maintaining an instance for a 501(c)(3), I just assume we'd have qualified).] This looks like an attempt at fixing their karma balance, but until I've reviewed the EULA I am pessimistic about the value of "allowing for some derivative works". And I don't really get the "allowing for some [..] contribution". <zyn>Is a patch that improves performance by 10% welcome but I'll have to pay them to make them accept a patch that improves performance by 100%?</zyn>
- timv 9y ago
  Disclosure: I work at Elastic, primarily on the X-Pack features in Elasticsearch > They introduced X-Pack by bundling it with the default distribution as a time-limited trial without explicitly stating that it is just a demo. I assume you're referring to the Elastic docker images, as I believe that's the only place where we've ever bundled X-Pack without any explicit opt-in (Our windows installer also includes X-Pack, but it's clearly marked as an optional & commercial component). We totally underestimated the confusion and difficulties that would cause, and it was fixed with the 6.0 release. The difficult was balancing the different needs of different users, with the constraints of how docker containers are typically managed. X-Pack requires a file-system level install, which is not something that docker users expect or want - an image should be built once and then be essentially immutable. No one likes to have to enter the container in order to install a new plugin. Since it was possible to disable X-Pack functionality, or install a free license for a subset of the features, it seemed like shipping the container with X-Pack pre-installed and letting users dial it back as needed, was the better option compare with shipping without X-Pack and forcing customers to reconfigure their container so that they could get the features that they needed. We didn't expect that there would be so many users for whom Docker was the primary/initial point of contact with our stack. We believed that we would be mostly working with users who already understood what X-Pack was and what they were getting. When it became clear that that wasn't true, we had to come up with a solution, which is what we shipped in the 6.0 release. We didn't want to change the behaviour in a minor release and cause more confusion for users who were relying on X-Pack being installed, so it wasn't simply a case of changing the way the images. The X-Pack licensing code was built on the premise that X-Pack was a plugin the was explicitly installed so that those who installed it would know what was going on. And when it was written, that was true. One of the consequences of that assumption was that it would automatically generate a trial license on start-up, and then you could install your own purchased license after the fact. In order to offer Docker containers that worked the way users would expect, we had to make changes to that so that X-Pack could be installed, but default to only enabling the free features, so since 6.0 we are now able to provide 3 different docker "flavours" - pure open source, basic (free license), platinum (trial license for paid features). We do want people to use X-Pack. We believe that the free (basic license) features offer something useful that users should know about, and the success of the company relies on users knowing that our commercial features exist, and being able to evaluate them and decide if that's something they want to purchase. But the docker situation was never intended to be a "bait-and-switch", it was just a problem that caught us by surprise and took some time to rectify. For historical accuracy, the inclusion of X-Pack in our docker images was not the introduction of X-Pack, it had been available for quite a long time before that and the underlying commercial IP was several years old before we started publishing Docker images.
hardwaresofton 9y ago
Yeah so has anyone actually tried to get ElasticSearch up and running lately? I just tried and had a terrible time, despite the fact that I was using ElasticSearch + Kibana, and it was dockerized, and it was on Kubernetes (there's more complexity, yes, but all those tools make deployment simpler once you understand them, not harder -- writing a pod resource config to get a thing running means I don't have to run around my system changing settings, I just put all of it in one place). XPack was just another stumbling block while trying to get everything running. The combination of lack of documentation, inconsistent/changed configuration (ENV vs YAML vs values that just don't exist anymore), breaking changes between versions that rendered Kibana completely useless, and the recent (?) removal of plugins that expose web APIs (so I couldn't use something like elastic-head. This is all in Kubernetes btw -- maybe it's just that I wasn't smart enough to get it done, but it's so easy to write functional (if not well-configured) configurations for other databases, I was at a loss for words when nothing I tried worked right. I got so angry trying to set up ElasticSearch that making a F/OSS competitor is now #2 on my list of projects-to-do-next. I'm sure the thought is naive but I need to find out for myself that there's no easier way. Imagine if the team behind Prometheus had focused on search instead of metrics? That's the kind of tool I want to use. A tool as focused, easy to start, clearly documented, and straightforward as prometheus.
- lobster_johnson 9y ago
  Elasticsearch is a mess. It's so full of historical warts. One major problem is that none of their documentation is actually reference documentation -- if you look for the formal schema (for things like mappings and the query DSL), the list of endpoints and their allowed parameters, the full list of settings etc., you won't find them listed anywhere. For example, does "keyword" mappings support the "enabled" property? What does the "index_options" setting actually do when combined with the "index" setting? Hard to tell any of this without trying them out. Turns out "dynamic_templates" mappings support any combination of the above, and will never complain about invalid combinations, whereas property mappings do. The whole environment variable vs Java property mess that you mention also exists. They do deserve credit for trying to clean it up. The last few releases have been pretty brutal in how they've been deprecating (and later removing) legacy features and tightening the semantics, the newest and most dramatic of which is the deprecation of multiple type mappings per index. And they've been pretty good at explaining what's going to happen. So the warts are getting fewer. On the flip side, you have to follow the release notes religiously if you want to keep up to speed, since each release now tends to remove a bunch of features or add strict validation where there previously was none, and it becomes harder to upgrade. (If you want an important bug fix that hasn't been backported, things could get expensive.) It's interesting how the Elasticsearch team let their focus be derailed by this new industry obsession with analytics and logs. It's not something ES was originally built for, and it turned out to be good at it mostly by accident. It's not terrible at it, but Elasticsearch shines the most for its original purpose, as a content index with rich full-text search capabilities. (Areas where it works less well include scaling edge cases such as high-cardinality aggregation buckets and high numbers of unique field names.) I wish they'd rather worked on things like joins and fixing the need for the "nested" object type, which is a ridiculous hack, but since those things aren't needed for analytics/logs, they haven't happened. (Pet peeve time: One problem that rarely gets mentioned is that Elasticsearch's "eventually consistent" model has two parts. There's the part where replicas may be out of sync with primaries, but there's also the problem that on each individual node, index operations don't become visible to queries right away, not until the next segment "refresh", which by default happens every second. There's no API to ask about the refresh state, so right not the only way for a read followed by a write to be consistent is to ask the write to wait for refresh (or force a refresh), which is the opposite of what you want; the wait should be on read, not write. Given that ES now has a sequence number associated with shards, I'm surprised they haven't tied those numbers together with refreshes so you can ask about which sequence number the index is currently "at".) So I think Elasticsearch is definitely ripe for disruption. I don't know of anything else that is able to compete at the moment, at least not in a single package; Solr isn't really in the same league.
abofh 9y ago
I'm a cynical es admin, but I'm pretty sure this is just a polite way of saying "you're still gonna need support" more than it is "it's good to go, run it yourself!" Between ES and RMQ, I find myself running software written before the cloud was big, but trying to be the drivers of the cloud -- those two pieces consume more resources, both physical and operational then the rest of our stack three times replicated, combined.
sscarduzio 9y ago
This thing of having the source "out there" (but not OSS) worries me and is very confusing for users because it creates dangerous grey areas. If you think like me, check out the actually FOSS(GPLv3) alternative to X-Pack security I created: https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin https://github.com/sscarduzio/elasticsearch-readonlyrest-plu...
cfontes 9y ago
Being honest X-PACK is just a pain for the small developer, it should be opt-in by default and only installed by the user not the other way around, I would probably be using some parts of it like that. I used to love Elastic even have a couple of small pull requests merged. I've felt bad for using such an awesome software for free and not giving back. But X-PACK changed that, I am now a bit afraid of putting my trust in the software and the first thing I do is uninstall X-PACK when I get my hands in an elastic cluster or github repo that carries it.
cik 9y ago
Saw this in my inbox yesterday and was excited... until I read it. Nothing changed. Ultimately Elastic is still looking for a revenue model beyond consulting that makes sense. Our company needed to build a bunch of this ourselves, simply because of the bundling cost. At the end of the day many, many organizations would love the ability to PURCHASE A SINGLE PLUGIN not all of x-pack. Ignoring the cost - the reality is that x-pack is useful in large enterprise, and a reasonably tiered model to develop against would help startups get going, and tie us in for the long-term. Instead, we built our own. The sheer number of times I've had direct conversations with people at Elastic (from CEO down) about this makes me cringe. We rely on Elastic. We currently have our own ecosystem around it. We're also going to (over time), probably have to open source bits and pieces, so that others don't have the same pain.
termez442 9y ago
Dear Elastic people reading this thread: We hope you wont do evil with your licence. If you don't do evil (e.g. write anything that could mess other people IP when working with Elasticsearch, prevent compatibility with other products, scare people from even looking at these codes or at the whole repository ) it will be.. a setep in the right direction. If you do evil it will cost you dearly in the end, with FUD about companies should be enormously weary of having anyone even look at those repositories. Hope we'll continue to cheer and celebrate your achievements.
rywalker 9y ago
Here we have a company that is simply opening up visibility of the code of their commercial offering, which will enhance customer relationships vs. keeping the code black box —— and it's observed as user-hostile? Open core isn't evil. Companies need to make money, and shouldn't have to choose between fully-open-source vs. closed-source. Fully-open-source companies have a really hard time of building a good company, because despite the great work they put out into the world, other enterprising cloud companies can build closed-source cloud services around it. For example, I've paid Compose.io and other companies real money around MongoDB, and never a cent to MongoDB the company. Same goes w/ Docker, who has struggled to build a great business because they open-sourced so much of their value. So yeah, Elastic is trying to make it more likely that you'll choose to buy a license, because the health of their business depends on getting paid customers.

159 points by Benfromparis 9y ago

Their licence is still proprietary, though. At the least people with a technical curiosity in X-Pack will be able to install it. Technically it's illegal, but Elastic probably doesn't care until the moment they start gaining momentum.

busterarm 9y ago
I wouldn't put it past the Elastic team to include some sort of phone-home mechanism here. Nor would I put it past them to consider legal demand letters as part of their sales funnel.
- Redsquare 9y ago
  Based on what exactly?
  - busterarm 9y ago
    Based on a previous experience being in their sales funnel and speaking to their reps at re:Invent. They have an incredible product that I can't live without, it's just incredibly painful trying to actually pay them money for something. Honestly, I think their real play here is getting community contributions to fix stuff that's very broken and also to edge out Siren Solutions. SearchGuard, Kibi and Sentinl are all _very_ good tools.
    - tylerhannan 9y ago
      I'm sorry to hear it was painful to pay us...I wish that wasn't the case. If you want, I'd be happy to talk more about it so that we can understand where we can improve. Feel free to email me tyler.hannan<at>elastic.co <disclaimer: I work at Elastic in Developer Relations>
      - arrayjumper 9y ago
        Not OP. One reason i find it is hard to pay you is because there is no way to buy your product without talking to someone. The pricing is not listed anywhere on your website.
        busterarm 9y ago
        That changed Jan 1 while were in the middle of figuring out what we were going to pay them -- then some X-Pack features disappeared from the "Gold" plan and the cost they quoted us was _way up_ from what we were evaluating. There actually still is a pricing page somewhere on the site, but you have to dig for it, and I don't think it's correct anymore.
- tylerhannan 9y ago
  In 6.3, telemetry (‘phone home’) is an opt-in only feature for both the free and paid components of X-Pack. We won’t be tracking any identification, so even if users opt-in to telemetry, there is no way to turn that into a sales strategy. Suing users is a pretty terrible business practice and not, at all, a funnel creation opportunity. Re. License, there are a few FAQs on https://www.elastic.co/products/x-pack/open#faq https://www.elastic.co/products/x-pack/open#faq that may help clear up any misunderstandings. Free features are free and governed by the Elastic EULA but are enabled by default. Paid features are available through an opt-in trial and, if they help solve a problem, can be purchased. <disclaimer: I work at Elastic in Developer Relations>
  - ohstopitu 9y ago
    and this trial can't be disabled (to use them for an unlimited amount of time) by someone by modifying your source because your license prohibits that? Am I correct in that assumption?
passionforcode 9y ago
I think open sourcing the code give opportunities to others to learn from the code and create own alternatives. What drive the community to involve in their software
- richdougherty 9y ago
  The main benefit may be that users can more easily use the code if they can read it. If you can see the code then you can understand how it works exactly, without relying on documentation to be accurate. You can also more easily debug your application if you have the source of third-party libraries. For writing clones or alternatives, it might be better not to read any proprietary code: https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design
  - passionforcode 9y ago
    You have a reason. That will be one of the main reasons. Sometimes I use open source projects to find a way to create something. Check many solutions can give an idea how to solve the puzzle. Make this projects open source can give me an idea to create plugins. Ofcourse still need think for yourself if it fits your needs, quality and if you not are struggle with possible copyrights.
continuations 9y ago
If their license is still proprietary what do they mean by "opening" the code?
- ryukafalz 9y ago
  They mean you can see the code. Note that they never say they're "open sourcing" it - it's not open source by the Open Source Definition. This is the license the X-Pack code is under: https://www.elastic.co/eula https://www.elastic.co/eula
  - bluecanary37 9y ago
    15 mentions of "open source" in the blog post and 12 more in the "Opening X-Pack" page, ignoring the other mentions of "open".

Jedd 9y ago

Eleven uses of the word free, all used in the gratis, not libre, sense of the word. 'Open' is in the same category for me now as 'cloud' -- too nebulous, too little consensus on what it actually means, often used in a 'don't worry about the details' hand-wavy way.

iramiller 9y ago

The important part is near the bottom, free components of X-Pack will no longer require the cumbersome registration process. Also of some concern is that extra care will be required to find a version of the software that is Apache 2.0 licensed as the proprietary licensed X-Pack components will be included by default in the code base instead of as a separate install. --- "Also, X-Pack features will now be bundled into the default distribution. All free features are included and enabled by default and will never ‘expire’, and commercial features are opt-in via a trial license. The license for free features never expires,you no longer need to register to use these capabilities. In addition to this, an Apache 2.0-only distribution will be created for download."

btown 9y ago

This is incredibly user-hostile, and if you read this expecting this is an open-source release, consider yourself clickbaited. Want to contribute to the open-source version of Elasticsearch? Want to ensure you're running an open-source database without running a custom EULA past your legal department? Well, make sure that after 6.3 is released, you don't accidentally download the default distribution, or browse the official Github repository, or clone from Github, as all of those will have components subject to an as-yet-unreleased, non-OSI-approved EULA (see the final paragraph in https://www.elastic.co/products/x-pack/open https://www.elastic.co/products/x-pack/open), which may have clauses that trigger liability to Elastic if you accidentally flip a switch or look at the wrong thing. (I am not a lawyer, this is not legal advice, but I shouldn't have to be to know if I can use software freely, and I can only guess worst-case scenarios because the terms aren't released.) If Elastic were truly committed to transparency, they would release the terms of their new EULA immediately, provide a way to access the head of the open-source portion of the repository without accepting the EULA, and make it clear what the pricing ramifications are if one opts into certain flags or reads certain files in a commercial capacity. (Pricing requires a custom quote, but a quick Google search suggests that the lowest tier in which REST and node-to-node communications are encrypted begins in the tens of thousands of dollars - not the kind of thing you want to take lightly.) Now more than ever, it's important that projects like https://github.com/floragunncom/search-guard https://github.com/floragunncom/search-guard , a free and open-source (Apache 2.0) alternative to some of these X-Pack components, are supported, and that the word is spread around them. (EDIT: Yes, they have commercial components, but those are in a separate repository, so you avoid any entanglements. Why isn't Elastic doing the same?) And hopefully, if Elastic doesn't maintain a fully-OSS repo, someone will create and maintain a friendly fully-OSS fork that can be combined with third-party software to carry on the legacy of this otherwise excellent database in the right way.

tylerhannan 9y ago
One of the awesome things about our products is that people care deeply about them. There is one thing, in particular, I want to address. 'If Elastic were truly committed to transparency, they would release the terms of their new Eula immediately...' We are working on it. This change is not one that was taken lightly and we have to ensure we have the right language for when we do release the license, etc. We will put it online as soon as we are able. We are committed to transparency and, also, we are committed to our users. Yes, we are asking for a fair bit of trust...and I hope we continue to prove ourselves worthy of that trust. <disclaimer: I work at Elastic in Developer Relations>

jlg23 9y ago

Background: They introduced X-Pack by bundling it with the default distribution as a time-limited trial without explicitly stating that it is just a demo. People who did the update were bitten weeks later when it just stopped working because the demo license had expired. [Documentation of this was ridiculously bad and I only learned through this post that there apparently was some way to get a free license (maintaining an instance for a 501(c)(3), I just assume we'd have qualified).] This looks like an attempt at fixing their karma balance, but until I've reviewed the EULA I am pessimistic about the value of "allowing for some derivative works". And I don't really get the "allowing for some [..] contribution". <zyn>Is a patch that improves performance by 10% welcome but I'll have to pay them to make them accept a patch that improves performance by 100%?</zyn>

timv 9y ago
Disclosure: I work at Elastic, primarily on the X-Pack features in Elasticsearch > They introduced X-Pack by bundling it with the default distribution as a time-limited trial without explicitly stating that it is just a demo. I assume you're referring to the Elastic docker images, as I believe that's the only place where we've ever bundled X-Pack without any explicit opt-in (Our windows installer also includes X-Pack, but it's clearly marked as an optional & commercial component). We totally underestimated the confusion and difficulties that would cause, and it was fixed with the 6.0 release. The difficult was balancing the different needs of different users, with the constraints of how docker containers are typically managed. X-Pack requires a file-system level install, which is not something that docker users expect or want - an image should be built once and then be essentially immutable. No one likes to have to enter the container in order to install a new plugin. Since it was possible to disable X-Pack functionality, or install a free license for a subset of the features, it seemed like shipping the container with X-Pack pre-installed and letting users dial it back as needed, was the better option compare with shipping without X-Pack and forcing customers to reconfigure their container so that they could get the features that they needed. We didn't expect that there would be so many users for whom Docker was the primary/initial point of contact with our stack. We believed that we would be mostly working with users who already understood what X-Pack was and what they were getting. When it became clear that that wasn't true, we had to come up with a solution, which is what we shipped in the 6.0 release. We didn't want to change the behaviour in a minor release and cause more confusion for users who were relying on X-Pack being installed, so it wasn't simply a case of changing the way the images. The X-Pack licensing code was built on the premise that X-Pack was a plugin the was explicitly installed so that those who installed it would know what was going on. And when it was written, that was true. One of the consequences of that assumption was that it would automatically generate a trial license on start-up, and then you could install your own purchased license after the fact. In order to offer Docker containers that worked the way users would expect, we had to make changes to that so that X-Pack could be installed, but default to only enabling the free features, so since 6.0 we are now able to provide 3 different docker "flavours" - pure open source, basic (free license), platinum (trial license for paid features). We do want people to use X-Pack. We believe that the free (basic license) features offer something useful that users should know about, and the success of the company relies on users knowing that our commercial features exist, and being able to evaluate them and decide if that's something they want to purchase. But the docker situation was never intended to be a "bait-and-switch", it was just a problem that caught us by surprise and took some time to rectify. For historical accuracy, the inclusion of X-Pack in our docker images was not the introduction of X-Pack, it had been available for quite a long time before that and the underlying commercial IP was several years old before we started publishing Docker images.

hardwaresofton 9y ago

Yeah so has anyone actually tried to get ElasticSearch up and running lately? I just tried and had a terrible time, despite the fact that I was using ElasticSearch + Kibana, and it was dockerized, and it was on Kubernetes (there's more complexity, yes, but all those tools make deployment simpler once you understand them, not harder -- writing a pod resource config to get a thing running means I don't have to run around my system changing settings, I just put all of it in one place). XPack was just another stumbling block while trying to get everything running. The combination of lack of documentation, inconsistent/changed configuration (ENV vs YAML vs values that just don't exist anymore), breaking changes between versions that rendered Kibana completely useless, and the recent (?) removal of plugins that expose web APIs (so I couldn't use something like elastic-head. This is all in Kubernetes btw -- maybe it's just that I wasn't smart enough to get it done, but it's so easy to write functional (if not well-configured) configurations for other databases, I was at a loss for words when nothing I tried worked right. I got so angry trying to set up ElasticSearch that making a F/OSS competitor is now #2 on my list of projects-to-do-next. I'm sure the thought is naive but I need to find out for myself that there's no easier way. Imagine if the team behind Prometheus had focused on search instead of metrics? That's the kind of tool I want to use. A tool as focused, easy to start, clearly documented, and straightforward as prometheus.

lobster_johnson 9y ago
Elasticsearch is a mess. It's so full of historical warts. One major problem is that none of their documentation is actually reference documentation -- if you look for the formal schema (for things like mappings and the query DSL), the list of endpoints and their allowed parameters, the full list of settings etc., you won't find them listed anywhere. For example, does "keyword" mappings support the "enabled" property? What does the "index_options" setting actually do when combined with the "index" setting? Hard to tell any of this without trying them out. Turns out "dynamic_templates" mappings support any combination of the above, and will never complain about invalid combinations, whereas property mappings do. The whole environment variable vs Java property mess that you mention also exists. They do deserve credit for trying to clean it up. The last few releases have been pretty brutal in how they've been deprecating (and later removing) legacy features and tightening the semantics, the newest and most dramatic of which is the deprecation of multiple type mappings per index. And they've been pretty good at explaining what's going to happen. So the warts are getting fewer. On the flip side, you have to follow the release notes religiously if you want to keep up to speed, since each release now tends to remove a bunch of features or add strict validation where there previously was none, and it becomes harder to upgrade. (If you want an important bug fix that hasn't been backported, things could get expensive.) It's interesting how the Elasticsearch team let their focus be derailed by this new industry obsession with analytics and logs. It's not something ES was originally built for, and it turned out to be good at it mostly by accident. It's not terrible at it, but Elasticsearch shines the most for its original purpose, as a content index with rich full-text search capabilities. (Areas where it works less well include scaling edge cases such as high-cardinality aggregation buckets and high numbers of unique field names.) I wish they'd rather worked on things like joins and fixing the need for the "nested" object type, which is a ridiculous hack, but since those things aren't needed for analytics/logs, they haven't happened. (Pet peeve time: One problem that rarely gets mentioned is that Elasticsearch's "eventually consistent" model has two parts. There's the part where replicas may be out of sync with primaries, but there's also the problem that on each individual node, index operations don't become visible to queries right away, not until the next segment "refresh", which by default happens every second. There's no API to ask about the refresh state, so right not the only way for a read followed by a write to be consistent is to ask the write to wait for refresh (or force a refresh), which is the opposite of what you want; the wait should be on read, not write. Given that ES now has a sequence number associated with shards, I'm surprised they haven't tied those numbers together with refreshes so you can ask about which sequence number the index is currently "at".) So I think Elasticsearch is definitely ripe for disruption. I don't know of anything else that is able to compete at the moment, at least not in a single package; Solr isn't really in the same league.

abofh 9y ago

I'm a cynical es admin, but I'm pretty sure this is just a polite way of saying "you're still gonna need support" more than it is "it's good to go, run it yourself!" Between ES and RMQ, I find myself running software written before the cloud was big, but trying to be the drivers of the cloud -- those two pieces consume more resources, both physical and operational then the rest of our stack three times replicated, combined.

sscarduzio 9y ago

This thing of having the source "out there" (but not OSS) worries me and is very confusing for users because it creates dangerous grey areas. If you think like me, check out the actually FOSS(GPLv3) alternative to X-Pack security I created: https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin https://github.com/sscarduzio/elasticsearch-readonlyrest-plu...

cfontes 9y ago

Being honest X-PACK is just a pain for the small developer, it should be opt-in by default and only installed by the user not the other way around, I would probably be using some parts of it like that. I used to love Elastic even have a couple of small pull requests merged. I've felt bad for using such an awesome software for free and not giving back. But X-PACK changed that, I am now a bit afraid of putting my trust in the software and the first thing I do is uninstall X-PACK when I get my hands in an elastic cluster or github repo that carries it.

cik 9y ago

Saw this in my inbox yesterday and was excited... until I read it. Nothing changed. Ultimately Elastic is still looking for a revenue model beyond consulting that makes sense. Our company needed to build a bunch of this ourselves, simply because of the bundling cost. At the end of the day many, many organizations would love the ability to PURCHASE A SINGLE PLUGIN not all of x-pack. Ignoring the cost - the reality is that x-pack is useful in large enterprise, and a reasonably tiered model to develop against would help startups get going, and tie us in for the long-term. Instead, we built our own. The sheer number of times I've had direct conversations with people at Elastic (from CEO down) about this makes me cringe. We rely on Elastic. We currently have our own ecosystem around it. We're also going to (over time), probably have to open source bits and pieces, so that others don't have the same pain.

termez442 9y ago

Dear Elastic people reading this thread: We hope you wont do evil with your licence. If you don't do evil (e.g. write anything that could mess other people IP when working with Elasticsearch, prevent compatibility with other products, scare people from even looking at these codes or at the whole repository ) it will be.. a setep in the right direction. If you do evil it will cost you dearly in the end, with FUD about companies should be enormously weary of having anyone even look at those repositories. Hope we'll continue to cheer and celebrate your achievements.

rywalker 9y ago

Here we have a company that is simply opening up visibility of the code of their commercial offering, which will enhance customer relationships vs. keeping the code black box —— and it's observed as user-hostile? Open core isn't evil. Companies need to make money, and shouldn't have to choose between fully-open-source vs. closed-source. Fully-open-source companies have a really hard time of building a good company, because despite the great work they put out into the world, other enterprising cloud companies can build closed-source cloud services around it. For example, I've paid Compose.io and other companies real money around MongoDB, and never a cent to MongoDB the company. Same goes w/ Docker, who has struggled to build a great business because they open-sourced so much of their value. So yeah, Elastic is trying to make it more likely that you'll choose to buy a license, because the health of their business depends on getting paid customers.