4 ms·

> Tired Of High Cost And Complexity Of Your Log Systems Is this really the main thing ElasticSearch is used for now? In 2016, I perhaps naively chose ElasticSe

> Tired Of High Cost And Complexity Of Your Log Systems Is this really the main thing ElasticSearch is used for now? In 2016, I perhaps naively chose ElasticSearch (via AWS's managed service) for an actual user-facing search feature. That service is still running, and now I'm looking to move it to self-hosted, but didn't want to actually run ElasticSearch (or more likely now, OpenSearch). So I looked at the ZincSearch home page, and the quoted heading seems odd.

dimitar 4y ago
Well everyone has logs, but not everyone has user-facing search features. I personally wouldn't recommend self-hosti gElasticSearch for logs; it's such a commodity now that it cheaper to get something like Datadog or another manage service (but do check the pricing,the last time I checked the managed logging service from Azure was very expensive for what it did).
marktangotango 4y ago
If you're on the jvm, lucene is actually pretty trivial to embed. The complexity comes around representing the data to be seached as "documents", ingesting the documents to be searched, reimporting when they change, and lucene's goofy query syntax.
- udp 4y ago
  And Solr wraps it reasonably nicely over HTTP if you’re not on the JVM.
  - dotancohen 4y ago
    As does Elastic Search.
in38years 4y ago
ElasticSearch is definitely used for user facing search too! In the WordPress community there are several integrations such as SearchPress (https://github.com/alleyinteractive/searchpress https://github.com/alleyinteractive/searchpress) and ElasticPress (https://github.com/10up/ElasticPress https://github.com/10up/ElasticPress)
jillesvangurp 4y ago
Elasticsearch is free as in beer and you can use the properly OSS Opensearch as an alternative. So, I don't really get the cost argument., As for hardware cost needed to run Elasticsearch/Opensearch, it scales down as well as it scales up these days. I have a couple of ~70$/month clusters with millions of documents in each. Not a problem that I need to fix. I can scale them up to cost a lot more and handle billions of documents easily. That's in Elastic Cloud, which would be the expensive option. If you go bare metal, you can get way more bang for your buck obviously. A couple of Xeon machines with some decent amount of CPU/Memory/SSD go a long way. As for complexity, they make it pretty easy to get started. What you do beyond that is kind of a luxury problem. You get all of these features out of the box but that doesn't mean you have to use them. But it's nice to have the option when you do need them as opposed to running into a brick wall because you are missing key features in these so-called drop in replacements. Depends on your needs. If you don't know the first thing about your search needs, keep it simple and go with mainstream solutions. Sign up with some managed opensearch/elasticsearch provider, get your cluster ready in a few minutes and start throwing some data at it. It's that simple. You can get it running in docker as well if you need something to develop against. It seems a bit of a rite of passage to be implementing your own search engine these days without being hindered too much by not being completely up to speed with the field of information retrieval and all the wonderful algorithms and sollutions. Guilty of that myself actually; I built a tf/idf implementation at some point. It's not that hard. From there to something that gets close to Elasticsearch/Opensearch/Solr/etc. takes a lot of work and skills, however. There are a few nice solutions out there but also lots that aren't so hot. I've not used Zincsearch so I'll reserve my judgement. However, their main pitch seems to be that they are cheaper than a thing that is free and partially compatible with it (for ingesting data only) but a bit lightweight (not so scalable, way less features). So maybe pay the difference of 0$ and get something without the limitations that come with being "lightweight"? Just saying. Not a great pitch if you have to name your main competitor in it. Kind of emphasizes that you are trying to catch up rather than disrupt. Does it have any actual things it does better? That should normally be the center piece of a pitch.
xani_ 4y ago
Running ES for simple site is trivial. If you're not pushing hundreds of GB or hundred+ searches per second, ES/opensearch is just pretty much "just run it"
- mwcampbell 4y ago
  It sure didn't look easy when I looked at the OpenSearch docs a few months ago. Any pointers to an easy way of configuring OpenSearch (or ElasticSearch) when I only want to run it on one machine, and it will only be accessed by other processes on that same machine (so I can just access it via localhost)?
ckosidows 4y ago
I haven't used it for nearly a year now, but in my last role we used ES for user-facing search. We had to turn to documentation, forums and support for a lot of problems we faced and I got the impression ES was mostly being used for logs while user-facing search was the less common use case. Support didn't have answers for us on a number of occasions and online documentation was equally sparse. Kibana was killer, though.
deleted 4y ago
[deleted]
prabhatsharma 4y ago
Elasticsearch and ZincSearch both are general purpose search engines based on inverted indexes that can be used for app search as well as log search. In my stint at AWS I found that most companies used elasticsearch and anecdotally 90%+ users used it for log search. Log search was also the use case that I had in my mind when I built ZincSearch for my own needs and I found ES to be lacking. Here is a blog that I wrote on my motivation around building ZincSearch - https://prabhatsharma.in/blog/in-search-of-a-search-engine-beyond-elasticsearch-introducing-zinc/ https://prabhatsharma.in/blog/in-search-of-a-search-engine-b...

dimitar 4y ago

Well everyone has logs, but not everyone has user-facing search features. I personally wouldn't recommend self-hosti gElasticSearch for logs; it's such a commodity now that it cheaper to get something like Datadog or another manage service (but do check the pricing,the last time I checked the managed logging service from Azure was very expensive for what it did).

marktangotango 4y ago

If you're on the jvm, lucene is actually pretty trivial to embed. The complexity comes around representing the data to be seached as "documents", ingesting the documents to be searched, reimporting when they change, and lucene's goofy query syntax.

udp 4y ago
And Solr wraps it reasonably nicely over HTTP if you’re not on the JVM.
- dotancohen 4y ago
  As does Elastic Search.

in38years 4y ago

ElasticSearch is definitely used for user facing search too! In the WordPress community there are several integrations such as SearchPress (https://github.com/alleyinteractive/searchpress https://github.com/alleyinteractive/searchpress) and ElasticPress (https://github.com/10up/ElasticPress https://github.com/10up/ElasticPress)

jillesvangurp 4y ago

Elasticsearch is free as in beer and you can use the properly OSS Opensearch as an alternative. So, I don't really get the cost argument., As for hardware cost needed to run Elasticsearch/Opensearch, it scales down as well as it scales up these days. I have a couple of ~70$/month clusters with millions of documents in each. Not a problem that I need to fix. I can scale them up to cost a lot more and handle billions of documents easily. That's in Elastic Cloud, which would be the expensive option. If you go bare metal, you can get way more bang for your buck obviously. A couple of Xeon machines with some decent amount of CPU/Memory/SSD go a long way. As for complexity, they make it pretty easy to get started. What you do beyond that is kind of a luxury problem. You get all of these features out of the box but that doesn't mean you have to use them. But it's nice to have the option when you do need them as opposed to running into a brick wall because you are missing key features in these so-called drop in replacements. Depends on your needs. If you don't know the first thing about your search needs, keep it simple and go with mainstream solutions. Sign up with some managed opensearch/elasticsearch provider, get your cluster ready in a few minutes and start throwing some data at it. It's that simple. You can get it running in docker as well if you need something to develop against. It seems a bit of a rite of passage to be implementing your own search engine these days without being hindered too much by not being completely up to speed with the field of information retrieval and all the wonderful algorithms and sollutions. Guilty of that myself actually; I built a tf/idf implementation at some point. It's not that hard. From there to something that gets close to Elasticsearch/Opensearch/Solr/etc. takes a lot of work and skills, however. There are a few nice solutions out there but also lots that aren't so hot. I've not used Zincsearch so I'll reserve my judgement. However, their main pitch seems to be that they are cheaper than a thing that is free and partially compatible with it (for ingesting data only) but a bit lightweight (not so scalable, way less features). So maybe pay the difference of 0$ and get something without the limitations that come with being "lightweight"? Just saying. Not a great pitch if you have to name your main competitor in it. Kind of emphasizes that you are trying to catch up rather than disrupt. Does it have any actual things it does better? That should normally be the center piece of a pitch.

xani_ 4y ago

Running ES for simple site is trivial. If you're not pushing hundreds of GB or hundred+ searches per second, ES/opensearch is just pretty much "just run it"

mwcampbell 4y ago
It sure didn't look easy when I looked at the OpenSearch docs a few months ago. Any pointers to an easy way of configuring OpenSearch (or ElasticSearch) when I only want to run it on one machine, and it will only be accessed by other processes on that same machine (so I can just access it via localhost)?

ckosidows 4y ago

I haven't used it for nearly a year now, but in my last role we used ES for user-facing search. We had to turn to documentation, forums and support for a lot of problems we faced and I got the impression ES was mostly being used for logs while user-facing search was the less common use case. Support didn't have answers for us on a number of occasions and online documentation was equally sparse. Kibana was killer, though.

deleted 4y ago

[deleted]

prabhatsharma 4y ago

Elasticsearch and ZincSearch both are general purpose search engines based on inverted indexes that can be used for app search as well as log search. In my stint at AWS I found that most companies used elasticsearch and anecdotally 90%+ users used it for log search. Log search was also the use case that I had in my mind when I built ZincSearch for my own needs and I found ES to be lacking. Here is a blog that I wrote on my motivation around building ZincSearch - https://prabhatsharma.in/blog/in-search-of-a-search-engine-beyond-elasticsearch-introducing-zinc/ https://prabhatsharma.in/blog/in-search-of-a-search-engine-b...