4 ms·
I was considering to use ElasticSearch to replace my CouchDB indexes that are way too slow, memory hungry, and not optimized. But I read somewhere, can't rememb
by speedgoose 7y ago
I was considering to use ElasticSearch to replace my CouchDB indexes that are way too slow, memory hungry, and not optimized. But I read somewhere, can't remember the source, that ElasticSearch doesn't offer any guarantee that all your data will (eventually) be saved or returned when queried. Is that the case?
- AznHisoka 7y agoThe default refresh rate for ES is a minute or so (can’t remember the exact time). This means when you index a document, it won’t be returned when you search for it until a minute later when the index refreshes. But you can certainly change the configuration so it’s practically real-time. There will be some performance hit though.
- heipei 7y agofwiw, the default refresh interval is 1 second. You should manually increase it if your application allows it. For bulk-indexing you can think about disabling it outright. Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
- frant-hartm 7y agoMaybe you are thinking of Aphyr's analysis of elastic search where he showed that Elastic can lose indexed documents during network partitions: https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-... That has been done on relatively old version. Elastic documents known (and fixed) issues here https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... but I wouldn't trust this to the letter mainly because of their previous handling of such issues. Elastic is great as a search index, not as a primary database.
- speedgoose 7y agoThanks for your clarification. Not using it as a primary database is fine and I understand that it's not designed for that. But I need to trust the results when I query it. Is it designed for fuzzy searches on many documents when missing one or two documents has no consequences or can I trust that I will always retrieve all documents that should match my query ?
- frant-hartm 7y agoIn theory with correct settings you should get all (or be told that the results are partial, e.g when shard is unavailable or timed out). In practice there are bugs, which are notoriously difficult to find and reproduce when you go distributed. Jepsen is closest to that I know about.
- speedgoose 7y agoThanks again. Distributed systems are difficults. I think I should have gone with a old school PostgreSQL.
- CameronNemo 7y agoThey have done quite a bit of work to improve resiliency. You can learn more about the status of this work here: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... TL;DR it is still probably not ideal as a primary data store.