3 ms·

About a year ago I set up an ES cluster to load Apache logs into as an experiment. Around a month later my boss asked if it was down. "Yeah, it looks like it,

About a year ago I set up an ES cluster to load Apache logs into as an experiment. Around a month later my boss asked if it was down. "Yeah, it looks like it, since you are using it let me upgrade it from an experiment to critical!" Since then we've been using it in more and more places, and are looking to see if it would be a good fit for storing some of our user data in. The big barrier right now is that we are running the version of SQLServer right before changed data capture became part of the standard edition, and that's the way we'd probably prefer to synchronize data into ES from our primary database. We have a couple home grown solutions to synchronizing secondary data sources, and we'd like to get out of that business. The SQL server is always going to be our source of truth, probably.

parthdesai 7y ago
>The SQL server is always going to be our source of truth, probably. ES is not supposed to be your source of truth.
- rhacker 7y ago
  Definitely agree - ES is designed more like a search appliance - it definitely should be pushed data from other databases that are the source of truth. https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/13 https://discuss.elastic.co/t/elasticsearch-as-a-primary-data...
- aliswe 7y ago
  In the early days at least, cosmosdb was built on es. Dunno about now though.
- ps101 7y ago
  Can you elaborate on the reasons for this?
  - lemcoe9 7y ago
    Here is a very good Quora answer on why you should never use ES as a central repository for data: https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my...
  - Izkata 7y ago
    Coming from experience with Solr, the answer is far simpler than the link in the sibling comment indicates: Data is processed and manipulated on import, and it's difficult - sometimes impossible, depending on the field config - to get that data back out in the format it was imported. Any changes to the schema, such as switching fields between indexed/not indexed/stored/not stored, requires reimporting the data to populate those fields, data which you're not likely to have if it was your primary store.
    - manigandham 7y ago
      Elasticsearch has a _source field [1] that stores the entire original document and is enabled by default. It's required to support features like highlighting in results. ES also has a reindex API that specifically makes use of this [2]. 1. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... 2. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...

parthdesai 7y ago

>The SQL server is always going to be our source of truth, probably. ES is not supposed to be your source of truth.

rhacker 7y ago
Definitely agree - ES is designed more like a search appliance - it definitely should be pushed data from other databases that are the source of truth. https://discuss.elastic.co/t/elasticsearch-as-a-primary-database/85733/13 https://discuss.elastic.co/t/elasticsearch-as-a-primary-data...
aliswe 7y ago
In the early days at least, cosmosdb was built on es. Dunno about now though.
ps101 7y ago
Can you elaborate on the reasons for this?
- lemcoe9 7y ago
  Here is a very good Quora answer on why you should never use ES as a central repository for data: https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my...
- Izkata 7y ago
  Coming from experience with Solr, the answer is far simpler than the link in the sibling comment indicates: Data is processed and manipulated on import, and it's difficult - sometimes impossible, depending on the field config - to get that data back out in the format it was imported. Any changes to the schema, such as switching fields between indexed/not indexed/stored/not stored, requires reimporting the data to populate those fields, data which you're not likely to have if it was your primary store.
  - manigandham 7y ago
    Elasticsearch has a _source field [1] that stores the entire original document and is enabled by default. It's required to support features like highlighting in results. ES also has a reindex API that specifically makes use of this [2]. 1. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... 2. https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...