6 ms·

When people say "Lightweight alternative" most of the time they are implying less features also, they are not lightweight just because

by ordiel 5y ago

jillesvangurp 5y ago
I would say a fraction of the features. If you don't need those features, that's fine of course. But in my experience, I end up using a lot of those features when dealing with real world customer requirements. I guess if you implement search for some website where search quality just doesn't matter, light weight is fine. For everything else, using proper solutions might be the wiser thing. In terms of performance you see a lot of comments that are stating X is faster than Y. I'd take such comments with a large grain of salt. Unless those lightweight alternatives actually do the same things you can't really compare the performance. It's not that hard to make indexing fast in Elasticsearch if you disable all the features that make it slower. Of course if you don't actually have those features it's going to be fast by default because it simply isn't doing anywhere close to the same things. Elasticsearch is actually pretty damn fast even when you do use its many features. The reason is that it relies on in memory caches, thread pools, etc. and that a lot of very smart people have been implementing and optimizing very efficient algorithms in Lucene for the last 25 years. Elasticsearch actually can run with as little as 256MB but of course you are not going to be able to cache a lot of data with that and performance will suffer accordingly. Mostly large heap sizes with Elasticsearch are all about using larger caches. It also relies on memory mapped files and OS file caches for that. That's what allows it to work with extremely large data sets and still provide query responses in milliseconds. There's no reason you wouldn't be able to do the same with a native implementation of course. But it would be naive to assume you'd end up using a lot less memory or CPU. Using that would be pretty core to matching performance and features. Not using that would make it pretty hard to get even close.
pjmlp 5y ago
Except every person has a different set of 10% features.
vosper 5y ago
Yeah. There's a lot that Elasticsearch can do. There are libraries and applications that can do bits of what ES can do, but no-one I'm aware of (and I watch this space) is even slightly close to building a genuine replacement (I'm not counting the AWS fork, for obvious reasons)
- jjeaff 5y ago
  Well, I suppose anything could be a genuine replacement depending on how many of the features of elasticsearch the user actually needs/uses.
- bpicolo 5y ago
  Where would you say Solr falls short? In terms of search - it's definitely the case that ES has expanded beyond that (metrics, apm...)
  - arafalov 5y ago
    I feel ES has been trying really hard to walk away from the features Solr is good at. While ES still supports multiple languages and custom tokenization chains and even custom pre-processing chains (somewhat equivalent to Solr's UpdateRequestProcessors), I felt that they were very deeply buried in the configuration, when I look at ES a year ago. ES is truly focusing on metrics and things and does have some features to make those use cases easier that Solr would probably need a lot of configuration/customization for. So, Solr is about search. ES is about a specific set of use cases that rely very heavily on search. And Lucidworks Fusion (commercial alternative to ES) is about big data and ML and full multi-tool pipeline on top of Solr.
    - znpy 5y ago
      ES wins hands down in search in my experience because the companies that were doing search stuff "before it was cool" were mostly running stuff like Autonomy DRE which work well but are expensive and proprietary. There is a huge market just by going after that kind of customers.
    - 5y ago
pageandrew 5y ago
but its written in Go
deleted 5y ago
[deleted]

When people say "Lightweight alternative" most of the time they are implying less features also, they are not lightweight just because

by ordiel 5y ago

When people say "Lightweight alternative" most of the time they are implying less features also, they are not lightweight just because

jillesvangurp 5y ago

I would say a fraction of the features. If you don't need those features, that's fine of course. But in my experience, I end up using a lot of those features when dealing with real world customer requirements. I guess if you implement search for some website where search quality just doesn't matter, light weight is fine. For everything else, using proper solutions might be the wiser thing. In terms of performance you see a lot of comments that are stating X is faster than Y. I'd take such comments with a large grain of salt. Unless those lightweight alternatives actually do the same things you can't really compare the performance. It's not that hard to make indexing fast in Elasticsearch if you disable all the features that make it slower. Of course if you don't actually have those features it's going to be fast by default because it simply isn't doing anywhere close to the same things. Elasticsearch is actually pretty damn fast even when you do use its many features. The reason is that it relies on in memory caches, thread pools, etc. and that a lot of very smart people have been implementing and optimizing very efficient algorithms in Lucene for the last 25 years. Elasticsearch actually can run with as little as 256MB but of course you are not going to be able to cache a lot of data with that and performance will suffer accordingly. Mostly large heap sizes with Elasticsearch are all about using larger caches. It also relies on memory mapped files and OS file caches for that. That's what allows it to work with extremely large data sets and still provide query responses in milliseconds. There's no reason you wouldn't be able to do the same with a native implementation of course. But it would be naive to assume you'd end up using a lot less memory or CPU. Using that would be pretty core to matching performance and features. Not using that would make it pretty hard to get even close.

pjmlp 5y ago

Except every person has a different set of 10% features.

vosper 5y ago

Yeah. There's a lot that Elasticsearch can do. There are libraries and applications that can do bits of what ES can do, but no-one I'm aware of (and I watch this space) is even slightly close to building a genuine replacement (I'm not counting the AWS fork, for obvious reasons)

jjeaff 5y ago
Well, I suppose anything could be a genuine replacement depending on how many of the features of elasticsearch the user actually needs/uses.
bpicolo 5y ago
Where would you say Solr falls short? In terms of search - it's definitely the case that ES has expanded beyond that (metrics, apm...)
- arafalov 5y ago
  I feel ES has been trying really hard to walk away from the features Solr is good at. While ES still supports multiple languages and custom tokenization chains and even custom pre-processing chains (somewhat equivalent to Solr's UpdateRequestProcessors), I felt that they were very deeply buried in the configuration, when I look at ES a year ago. ES is truly focusing on metrics and things and does have some features to make those use cases easier that Solr would probably need a lot of configuration/customization for. So, Solr is about search. ES is about a specific set of use cases that rely very heavily on search. And Lucidworks Fusion (commercial alternative to ES) is about big data and ML and full multi-tool pipeline on top of Solr.
  - znpy 5y ago
    ES wins hands down in search in my experience because the companies that were doing search stuff "before it was cool" were mostly running stuff like Autonomy DRE which work well but are expensive and proprietary. There is a huge market just by going after that kind of customers.
  - 5y ago

pageandrew 5y ago

but its written in Go

deleted 5y ago

[deleted]