4 ms·

The 'Recap' section has good advice to address the issue of data loss: My recommendations for Elasticsearch users are unchanged: store your data in a database

by w8rbt 11y ago

The 'Recap' section has good advice to address the issue of data loss: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.

machbio 11y ago
I do something similar but my data is not that important, as most of you guys work on - I have a very high rate of data being input from users for curating purposes in a bioinformatics lab..So I load the data onto a mongodb cluster and run the elasticsearch-river-mongodb [1] to keep the elasticsearch in sync with the mongodb.. I do not know whether it solves this problem - but I am happy that I have data safe in mongodb - any inputs regarding my method would be appreciated.. [1]https://github.com/richardwilly98/elasticsearch-river-mongodb.git https://github.com/richardwilly98/elasticsearch-river-mongod...
- benburton 11y ago
  Watch out, rivers are being deprecated: https://www.elastic.co/blog/deprecating_rivers https://www.elastic.co/blog/deprecating_rivers
- cwyers 11y ago
  "Data safe in Mongo" seems... less reassuring than I think you meant it to be: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
- nosequel 11y ago
  > I am happy that I have data safe in mongodb Oh no. So you are protecting your data from unsafe Elasticsearch by storing it in an even more unsafe MongoDB[1]? You should probably put a database that knows how to store data somewhere in your stack and learn to use that. [1]: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...

by w8rbt 11y ago

machbio 11y ago

I do something similar but my data is not that important, as most of you guys work on - I have a very high rate of data being input from users for curating purposes in a bioinformatics lab..So I load the data onto a mongodb cluster and run the elasticsearch-river-mongodb [1] to keep the elasticsearch in sync with the mongodb.. I do not know whether it solves this problem - but I am happy that I have data safe in mongodb - any inputs regarding my method would be appreciated.. [1]https://github.com/richardwilly98/elasticsearch-river-mongodb.git https://github.com/richardwilly98/elasticsearch-river-mongod...

benburton 11y ago
Watch out, rivers are being deprecated: https://www.elastic.co/blog/deprecating_rivers https://www.elastic.co/blog/deprecating_rivers
cwyers 11y ago
"Data safe in Mongo" seems... less reassuring than I think you meant it to be: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
nosequel 11y ago
> I am happy that I have data safe in mongodb Oh no. So you are protecting your data from unsafe Elasticsearch by storing it in an even more unsafe MongoDB[1]? You should probably put a database that knows how to store data somewhere in your stack and learn to use that. [1]: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...