4 ms·

Is ES one of these necessary evils? I have been looking for a product similar to it, but it seems ES is always recommended.

mooreds 7y ago
I saw a post about this and the other open source alternative was solr. Not sure about support/community though.
- ink_13 7y ago
  It's funny because under the hood, they're both Lucene.
  - mooreds 7y ago
    I guess that's the power of open source, right?
cbetti 7y ago
No, ES is an affordable entry level search engine in a very expensive product space. But there is nothing new in the standalone search engine space that I am aware of. Many engines have been gobbled up and integrated into larger product suites (see: Oracle Secure Enterprise Search).
33degrees 7y ago
I don’t find ES that bad, considering its power, but I haven’t had the best experience with the AWS service. Could be user error though....
- BoorishBears 7y ago
  I don't fault ES for this necessarily, but the biggest problem with ES is, it's easy to build something that looks like it works, right until it doesn't. This is a problem for most DBs. You put stuff in, you take stuff out, and it works... but you're putting it in the DB in a way that will make the queries you need nearly impossible to do within your real-world/production constraints (response time, resources required, etc.) The thing is ES is so different from most DBs, so you need to relearn how to do things right. And to make matters a little more difficult, all DBs have a lot of knobs, but ES just has way more knobs you need to know how to turn to get something that won't fall on it's face in production. I don't remember my experience with ES on AWS that well besides it was behind on major versions for a while at one point, and if you set up ES resource with CloudFormation... god help you if you need to roll them back. Using serverless with ES looked something like api-1 api-2-because-1-is-frozen-rolling-back-cloudformation-es-instances api-3-because-2-is-also-frozen-rolling-back-cloudformation-es-instances api-4-...
  - shiredude95 7y ago
    Have you tried elasticcloud? They solve a lot of access problems present in AWS Elasticsearch along with providing the most recent version for deployment.
    - Redsquare 7y ago
      Elasticcloud has terrible security features - they removed IP whitelisting.
      - t0mbstone 7y ago
        They do have user/password authentication though, which is something that barebones ElasticSearch doesn't normally have out of the box by default.
sidlls 7y ago
ES is one of those technologies that "everyone" goes to when they want any kind of basic text search indexing "at scale" (usually with a delusional notion of what their scaling needs actually are), because text search/indexing is tedious and hard at any scale that a basic DB query would fall over on (which is generally a much bigger scale than many want to admit).
falcolas 7y ago
The problem with ElasticSearch is that it does multiple things at once. It indexes documents. It stores documents. It searches indexes. It serves documents. The indexing and searching should be separated from the storage and serving at any kind of reasonable scale. But, ES makes it “just work” at the proof of concept and low volume stages, and you can’t easily back out of it when you reach its limitations.
- castillar76 7y ago
  For all that people (rightfully) give them grief for being expensive, this is something Splunk got right. You can put your entire Splunk infrastructure on one host: like stem cells, a single Splunk box can perform any of the required functions (ingest, index, search). But you can also separate out each layer into separate hosts and then cluster them, so you have clusters of ingesters feeding clusters of indexers, accessed by clusters of search heads. That should be the architecture ES aims for, where people are given the flexibility to take whichever function is currently a pain point and break it out to a separate set of nodes.
pas 7y ago
Well, ES spent many years polishing itself, but there is one new very rough kid on the block: https://github.com/toshi-search/Toshi/blob/master/README.md https://github.com/toshi-search/Toshi/blob/master/README.md

mooreds 7y ago

I saw a post about this and the other open source alternative was solr. Not sure about support/community though.

ink_13 7y ago
It's funny because under the hood, they're both Lucene.
- mooreds 7y ago
  I guess that's the power of open source, right?

cbetti 7y ago

No, ES is an affordable entry level search engine in a very expensive product space. But there is nothing new in the standalone search engine space that I am aware of. Many engines have been gobbled up and integrated into larger product suites (see: Oracle Secure Enterprise Search).

33degrees 7y ago

I don’t find ES that bad, considering its power, but I haven’t had the best experience with the AWS service. Could be user error though....

BoorishBears 7y ago
I don't fault ES for this necessarily, but the biggest problem with ES is, it's easy to build something that looks like it works, right until it doesn't. This is a problem for most DBs. You put stuff in, you take stuff out, and it works... but you're putting it in the DB in a way that will make the queries you need nearly impossible to do within your real-world/production constraints (response time, resources required, etc.) The thing is ES is so different from most DBs, so you need to relearn how to do things right. And to make matters a little more difficult, all DBs have a lot of knobs, but ES just has way more knobs you need to know how to turn to get something that won't fall on it's face in production. I don't remember my experience with ES on AWS that well besides it was behind on major versions for a while at one point, and if you set up ES resource with CloudFormation... god help you if you need to roll them back. Using serverless with ES looked something like api-1 api-2-because-1-is-frozen-rolling-back-cloudformation-es-instances api-3-because-2-is-also-frozen-rolling-back-cloudformation-es-instances api-4-...
- shiredude95 7y ago
  Have you tried elasticcloud? They solve a lot of access problems present in AWS Elasticsearch along with providing the most recent version for deployment.
  - Redsquare 7y ago
    Elasticcloud has terrible security features - they removed IP whitelisting.
    - t0mbstone 7y ago
      They do have user/password authentication though, which is something that barebones ElasticSearch doesn't normally have out of the box by default.

sidlls 7y ago

ES is one of those technologies that "everyone" goes to when they want any kind of basic text search indexing "at scale" (usually with a delusional notion of what their scaling needs actually are), because text search/indexing is tedious and hard at any scale that a basic DB query would fall over on (which is generally a much bigger scale than many want to admit).

falcolas 7y ago

The problem with ElasticSearch is that it does multiple things at once. It indexes documents. It stores documents. It searches indexes. It serves documents. The indexing and searching should be separated from the storage and serving at any kind of reasonable scale. But, ES makes it “just work” at the proof of concept and low volume stages, and you can’t easily back out of it when you reach its limitations.

castillar76 7y ago
For all that people (rightfully) give them grief for being expensive, this is something Splunk got right. You can put your entire Splunk infrastructure on one host: like stem cells, a single Splunk box can perform any of the required functions (ingest, index, search). But you can also separate out each layer into separate hosts and then cluster them, so you have clusters of ingesters feeding clusters of indexers, accessed by clusters of search heads. That should be the architecture ES aims for, where people are given the flexibility to take whichever function is currently a pain point and break it out to a separate set of nodes.

pas 7y ago

Well, ES spent many years polishing itself, but there is one new very rough kid on the block: https://github.com/toshi-search/Toshi/blob/master/README.md https://github.com/toshi-search/Toshi/blob/master/README.md