4 ms·
Others are sharing out of the box solutions. But I will say that many moons ago when I did actually write stuff for Mongo. The oplog was a god send. You can "t
by mindingdata 8y ago
Others are sharing out of the box solutions.
But I will say that many moons ago when I did actually write stuff for Mongo. The oplog was a god send. You can "tail" the oplog, and get every transaction in near real time. We used this for updating Elasticsearch indexes etc in what is basically realtime, without having to poll or modify existing code at all.
- sateesh 8y agoYes you can tail oplog, but in a sharded setup with multiple clusters this becomes a pain.
- sbr464 8y agoThe new change streams api resolved a lot of this, since it allows your to utilize the aggregation framework to subscribe to highly filtered/specific queries. The upcoming 4.0 release is expanded further.
- chiaolun 8y agoThe oplog is awesome! It provides an immutable record which is really useful - we materialize the oplog directly in Athena to get a time-travelling database for debugging purposes.
- kalendos 8y agoWould you mind sharing the process? I'm curious which flow you use to materialize the oplog in Athena/S3.
- Ankhers 8y agoMongo 3.6 introduced something called Change Streams[0] which is basically a safer way to tail the oplog. It is also supposed to work well in a sharded environment. [0] https://docs.mongodb.com/manual/changeStreams/ https://docs.mongodb.com/manual/changeStreams/