3 ms·

A lot of people advocate PostgreSQL as a replacement. Why not elasticsearch? It’s also a document store with a good history. At a previous job I shipped Mong

A lot of people advocate PostgreSQL as a replacement. Why not elasticsearch? It’s also a document store with a good history. At a previous job I shipped Mongo as the store (under AGPL) because at the time the replication story for PGSQL was too complex compared to mongo. But I reached a point where I was going to replace Mongo with elasticsearch and get rid of Solr at the same time.

reaperducer 7y ago
I may have a project coming up that will include a document store component. It's in-house, with the maximum number of simultaneous users under 15, so scalability that way isn't a concern. However, the number of documents is large, and for some documents previous versions will have to be stored, as well. With Mongo out, do you have any suggestions? Is PostgreSQL the best option here? I'm fairly new to this space.
- edoceo 7y ago
  I do like this, millions of record, prime fields in the table, then a jsonb column as the document-store. Just add a row for a new version. And extracting detail from jsonb is pretty easy, usable in SqL queries and materialized views
- kitotik 7y ago
  I just recently deployed a cockroachdb, and have been very happy with the results. For my use case, the JSONB columns are fantastic, and has much less deploy and admin overhead than Postgres.
McGlockenshire 7y ago
I asked myself this a few years ago and went to see who had thought of it first. At the time, the prevailing opinion was that this was a bad idea. Elasticsearch had no ACID data integrity guarantees at all at the time. Has this changed?
- craigching 7y ago
  My mongodb knowledge is probably a bit out-of-date here. It didn’t have them either when I was using it. Multi-document ACID would have been really useful, is that a feature now? That would be one data point against elasticsearch.
33degrees 7y ago
ElasticSearch has some resiliency issues, which are well documented: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... I've run into some weird issues in the past with data types changing by themselves, but it may have been user error: in any case being able to reindex from scratch fixed the problem.
- craigching 7y ago
  Just as an anecdote to the jepsen test findings, I’ve used elasticsearch at two jobs as a primary datasource and haven’t had data loss as mentioned in the tests. The tests are, at this point, pretty old and I’ve found that newer versions are pretty good at controlling data loss. As a counterpoint, mongo also still has some outstanding issues with jepsen tests, but they are working on being compliant. Honestly, as far as resiliency, I’m not sure either has solved the problem perfectly.
luizfelberti 7y ago
I would strongly advise against this. For a long time, the only reason ElasticSearch was not the ugly duckling of distributed systems was because MongoDB existed. It does not have a good history at all. There was a time that Elastic even claimed that it beat the CAP theorem, which always makes me chuckle when I'm reminded of it, and historically they also have been quite vague about the distributed properties (and shortcomings) of ES. It works fine as a secondary datastore when you would benefit from all of the implemented search functions, but I wouldn't trust it for consistency (so I guess in a way it is a replacement for mongo \s) Not only that but having operated ES clusters before, it's a tricky thing to manage, and depending on your workload they can be very resource hungry. PostgreSQL is very solid though, and will scale well for the majority of people. I highly recommend this over ES, not to mention it does have some search features too, you should check them out.
- craigching 7y ago
  Depends on the version of elasticsearch you’re running and the use-case. If you’re running a single index with a well-defined schema, I think elasticsearch works very well. If you need many indices and can’t have a defined scheme, then it can be tough. I work on a team at Target that provides a log aggregation product and we’re finding that a single index with a well-defined schema used to troubleshoot applications via logs is very easy to manage and scale. We support some legacy patterns like “index per application” that are a pain in the ass to manage. Just to provide some context, we manage about 40 clusters that range from a few data nodes to about 160 data nodes with the majority falling in the 16-60 node ballpark. We don’t have a problem with data loss due to elasticsearch, generally speaking.

reaperducer 7y ago

I may have a project coming up that will include a document store component. It's in-house, with the maximum number of simultaneous users under 15, so scalability that way isn't a concern. However, the number of documents is large, and for some documents previous versions will have to be stored, as well. With Mongo out, do you have any suggestions? Is PostgreSQL the best option here? I'm fairly new to this space.

edoceo 7y ago
I do like this, millions of record, prime fields in the table, then a jsonb column as the document-store. Just add a row for a new version. And extracting detail from jsonb is pretty easy, usable in SqL queries and materialized views
kitotik 7y ago
I just recently deployed a cockroachdb, and have been very happy with the results. For my use case, the JSONB columns are fantastic, and has much less deploy and admin overhead than Postgres.

McGlockenshire 7y ago

I asked myself this a few years ago and went to see who had thought of it first. At the time, the prevailing opinion was that this was a bad idea. Elasticsearch had no ACID data integrity guarantees at all at the time. Has this changed?

craigching 7y ago
My mongodb knowledge is probably a bit out-of-date here. It didn’t have them either when I was using it. Multi-document ACID would have been really useful, is that a feature now? That would be one data point against elasticsearch.

33degrees 7y ago

ElasticSearch has some resiliency issues, which are well documented: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... I've run into some weird issues in the past with data types changing by themselves, but it may have been user error: in any case being able to reindex from scratch fixed the problem.

craigching 7y ago
Just as an anecdote to the jepsen test findings, I’ve used elasticsearch at two jobs as a primary datasource and haven’t had data loss as mentioned in the tests. The tests are, at this point, pretty old and I’ve found that newer versions are pretty good at controlling data loss. As a counterpoint, mongo also still has some outstanding issues with jepsen tests, but they are working on being compliant. Honestly, as far as resiliency, I’m not sure either has solved the problem perfectly.

luizfelberti 7y ago

I would strongly advise against this. For a long time, the only reason ElasticSearch was not the ugly duckling of distributed systems was because MongoDB existed. It does not have a good history at all. There was a time that Elastic even claimed that it beat the CAP theorem, which always makes me chuckle when I'm reminded of it, and historically they also have been quite vague about the distributed properties (and shortcomings) of ES. It works fine as a secondary datastore when you would benefit from all of the implemented search functions, but I wouldn't trust it for consistency (so I guess in a way it is a replacement for mongo \s) Not only that but having operated ES clusters before, it's a tricky thing to manage, and depending on your workload they can be very resource hungry. PostgreSQL is very solid though, and will scale well for the majority of people. I highly recommend this over ES, not to mention it does have some search features too, you should check them out.

craigching 7y ago
Depends on the version of elasticsearch you’re running and the use-case. If you’re running a single index with a well-defined schema, I think elasticsearch works very well. If you need many indices and can’t have a defined scheme, then it can be tough. I work on a team at Target that provides a log aggregation product and we’re finding that a single index with a well-defined schema used to troubleshoot applications via logs is very easy to manage and scale. We support some legacy patterns like “index per application” that are a pain in the ass to manage. Just to provide some context, we manage about 40 clusters that range from a few data nodes to about 160 data nodes with the majority falling in the 16-60 node ballpark. We don’t have a problem with data loss due to elasticsearch, generally speaking.