4 ms·

I've heard that Elasticsearch has had some durability issues in the past and the recommendation has been to have a more reliable source of truth (e.g. an SQL da

by vore 7y ago

I've heard that Elasticsearch has had some durability issues in the past and the recommendation has been to have a more reliable source of truth (e.g. an SQL database) and periodically import data into Elasticsearch for indexing. Is this still a problem in practice, or is storing data directly in Elasticsearch like this still unsafe?

softwaredoug 7y ago
I definitely think it shouldn’t be your source of truth. However, I think as a public facing content engine it’s fine. I know of one major e-commerce brand that does this for products. This works great for their use case, given how easily Elasticsearch lets you filter, sort, search pretty seemlessly. But they can always rebuild Elasticsearch from another system.
colechristensen 7y ago
It is, at least, a very much diminished issue in my experience going from 2.x to 7.x Somewhere around 5 or 6 the tides turned. What was a big issue is how easy it was to overload the cluster and how data would be lost in that state. To be safe you should have regular backups anyway. I wouldn't trust ES to handle very critical data which you wouldn't ever want to lose a bit of, but I have much less worry these days about the operational overhead of fixing an cluster because failure in experience is just much less likely. It is still though, something which is less "set and forget" like many sql engines can be in less critical or intense workloads. You would want someone in your org to really know ES.
jillesvangurp 7y ago
They re-implemented the clustering algorithm for v7 and fixed numerous bugs in between v1/v2 and v7. Including essentially all of the ones reported in the infamous call me maybe article (https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0 https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0). However, it's not a database. I've actually abused it as such and it's fine. You get optimistic locking but no real transactions. Search is eventually consistent unless you call _refresh (but you shouldn't), etc. Bearing in mind it is not a database, it is not intended to be used as a database, and probably will never be a database, it actually works fine as a database provided you don't do a lot of updates (write model is append only). If safety is a big concern for you, obviously use something else. But for a blog it's completely fine.

by vore 7y ago

softwaredoug 7y ago

I definitely think it shouldn’t be your source of truth. However, I think as a public facing content engine it’s fine. I know of one major e-commerce brand that does this for products. This works great for their use case, given how easily Elasticsearch lets you filter, sort, search pretty seemlessly. But they can always rebuild Elasticsearch from another system.

colechristensen 7y ago

It is, at least, a very much diminished issue in my experience going from 2.x to 7.x Somewhere around 5 or 6 the tides turned. What was a big issue is how easy it was to overload the cluster and how data would be lost in that state. To be safe you should have regular backups anyway. I wouldn't trust ES to handle very critical data which you wouldn't ever want to lose a bit of, but I have much less worry these days about the operational overhead of fixing an cluster because failure in experience is just much less likely. It is still though, something which is less "set and forget" like many sql engines can be in less critical or intense workloads. You would want someone in your org to really know ES.

jillesvangurp 7y ago

They re-implemented the clustering algorithm for v7 and fixed numerous bugs in between v1/v2 and v7. Including essentially all of the ones reported in the infamous call me maybe article (https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0 https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0). However, it's not a database. I've actually abused it as such and it's fine. You get optimistic locking but no real transactions. Search is eventually consistent unless you call _refresh (but you shouldn't), etc. Bearing in mind it is not a database, it is not intended to be used as a database, and probably will never be a database, it actually works fine as a database provided you don't do a lot of updates (write model is append only). If safety is a big concern for you, obviously use something else. But for a blog it's completely fine.