6 ms·
Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Goo
by lm28469 7y ago
Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene.
I guess you don't see much demand because for a lot of use cases the basic setups are good enough.
- tombert 7y agoAbout seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was stupid to do on someone else's time, and called the client to tell them that I'm moving to Solr, and I got the project done before-schedule as a result. ==== I think for 99% of usecases (involving search), Lucene/Solr/ES is perfectly fine. However, I do absolutely hate that some companies have decided to make it their primary database. EDIT: I just want to make it clear, I think it's totally valid to try and reinvent Solr for fun, or if that's something you're paid specifically to do; nothing is perfect, and I am actually a big fan of the "if it works, break it and make it better!" mentality.
- eivarv 7y agoAgreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. TLDR: Lucene < 7.5 won't merge segments larger than 5GB (default) unless they accumulate 50% deletions. Delivering a conference talk [1] later this year about it. [0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-and-gdpr https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-... [1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-53166ada9f08 https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...
- bryanrasmussen 7y agoI think the instructor's response in the linked article is a reasonable defense, you don't really know that data is deleted all the way down to the file level. It is just marked as deleted and could be retrieved by someone clever enough to do so. At some point in the future it will be really deleted. I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it was not a good enough deletion. Finally I have to ask this part: assuming ES is not your primary database, how does this get around the GDPR issues? If someone wants their data erased you are supposed to erase it from wherever you store data, I suppose this means ES when it indexes a primary store and finds it has deleted data actually deletes it but if it is told to delete something it keeps it around?
- rickmode 7y agoWhen using ES for indexing and not the primary store, you can (and should) periodically fully reindex the data set. You can use a blue / green pattern — create a new index then swap from the old one to the new one. ES supports aliases, making this swapping transparent to the apps using the index. Now you have more options. If it is easy to delete specific users from the primary database, the deleted users will naturally disappear during the next ES reindex. Edit: The old index is deleted at the file system level. If the reindexing occurs daily or weekly, perhaps this will satisfy GDPR. There are other good reason to not use ES as the primary data store. First, it isn’t entirely reliable. It’s good and I’ve never seen a corruption, but ES and Lucene’s history isn’t as a reliable database. Second, if you want to change how you index, it is a bit easier to do if the source data is outside of ES.
- bryanrasmussen 7y agothanks, I wasn't arguing that using ES as primary was good. Just don't necessarily see the GDPR argument as being a reasonable one. Although I've seen some startups using Mongo as primary and have to wonder if there would be that big a difference in using ES at that point (not a Mongo dig as I've kept away from it for various reasons)
- 7y ago
- penagwin 7y ago> "if it works, break it and make it better!" Fix it 'til it breaks is what I always say :D Needless to say my 3D printer had a lot of down-time haha.
- twic 7y ago"If it ain't broke, open it up and find out what makes it so bloody special" - i think that's wisdom from the BOFH.
- aliswe 7y agoI can chime in here that lucene- based solutions are sufficient almost always, for a purely frontend, js-based fuzzy search engine check out fusejs. https://fusejs.io/ https://fusejs.io/
- deleted 7y ago[deleted]
- hiram112 7y agoCan you explain a bit why ES isn't a good solution for storing data itself? I inherited a legacy Mongo solution, and all the data is duplicated and indexed in ES, so I've always wondered why we're using both. Mongo has none of the SQL capabilities that would make my life easier, and the types of queries allowed by Mongo could be done with ES. What are the negatives of ES alone?
- manigandham 7y agoIt's not reliable: https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore/answer/Mani-Gandham https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my... The v7 upgrade to a new cluster protocol (zen2) has improved things but overall the system has a long history of losing or destroying data. It's better to have a primary OLTP system that's ACID and reliable while using ES as the secondary search source. You can also remove the _source field if you just need matches without the original content. It's common to see pattern used with a relational database since, as you can see, Mongo doesn't buy you much else as another document-store.
- manigandham 7y agoFollow up: MongoDB is adding full-text search capabilities: https://www.youtube.com/watch?v=4QUGWnz-XaA https://www.youtube.com/watch?v=4QUGWnz-XaA
- hiram112 7y agoThank you very much. It seemed like we were just duplicating a bunch of scheme-less json for no reason, but if ES can lose data, that's probably not a good idea.
- busterarm 7y agoThere's folks working on Bleve (written in Go) and developers that I work with want to use it (we use Elasticsearch heavily), but as I've told everyone like you just did, Lucene has a 20 year head start. Thing is, there's heavy demand for something more performant than Elasticsearch, so eventually the market will provide. Meanwhile, Redis Enterprise is trying to grab some 'share with RediSearch, which has some severe caveats IMO that make it not a great fit for most.
- bpicolo 7y agoTantivy is an interesting project I'd point to in this space: https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy That said - it's effectively Lucene rewritten in rust, so the main win is some performance gains. Lucene has spent a ton of time getting the details right, and it's unlikely we'll see an order of magnitude of innovation in that particular space. At the higher level for querying / query understanding it feels like there's still more technological room to grow vs the lower level details.
- pornel 7y agoSeconded Tantivy. Very easy to set up and maintain, and faast.
- fulmicoton 7y agoTantivy main dev here. Thanks for the free marketing :) It is not exactly a port but yeah. tantivy is strongly inspired from Lucene. > Lucene has spent a ton of time getting the details right, and it's unlikely we'll see an order of magnitude of innovation in that particular space. Have you checked out the perf gain in Lucene 8.0 ? Block-WAND proved you wrong.
- bpicolo 7y agoI suppose I could have phrased that better. I appreciate the correction. I mostly mean to say that the functionality is really impressive today, and serves its use case very well for the intended target of lower level search primitives. Tantivy is a cool project, but I have to say the part I love most about it is your blog posts on it. They're a great introduction for people who are unfamiliar with the underlying tech of search engines.
- deathtrader666 7y agoWonder what DuckDuckGo might have built for its search..
- ksec 7y ago>Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. Vespa [1] would like to have a word with you. [1]https://vespa.ai https://vespa.ai
- softliving 7y agomove on! IMO donated tool, Yahoo can't maintain it, give it to community reduce the cost. If only it was a new project which wanted to address search in a different way.
- ksec 7y agoIf anything Vespa has actually gotten faster in development since it Open Sourced.
- jerska 7y agoI’ll just chime in to say that Algolia runs on an home-made C++ engine. [I work at Algolia]