3 ms·
Take a look at Kafka. When your backend pushes data to MongoDB, have it produce a Kafka message that contains serialized Elasticsearch JSON documents and then u
by FBISurveillance 8y ago
Take a look at Kafka. When your backend pushes data to MongoDB, have it produce a Kafka message that contains serialized Elasticsearch JSON documents and then use either a bespoke consumer, or kafka-connect-elasticsearch, or logstash-input-kafka to consume those documents and bulk-insert them into Elasticsearch.
Bonus points: it will let you have a standby Elasticsearch cluster in another region you could failover to when needed, perform Elasticsearch cluster upgrades/maintenance without losing data, and more.
Depending on your use case this could be an overkill so YMMV.
- anikdas 8y agoI could definitely do that. But my original intention was to use a standalone application which can get the job done. For example when the data is updated/deleted instead of insertion I need to write my own logic. Any way great suggestion. Will definitely keep in consideration.