3 ms·
Tailing the MongoDB Replica Set Oplog with Scala and Akka Stream
- altendo 11y agoThis is useful for any collection that is also a capped collection (that is, the collection has only a certain amount of finite space allocated to it), not just the oplog. In fact, the oplog is just a special capped collection, hence why tailable cursors are available for it. Also note that non-capped collections don't support tailable cursors, so this won't work. Documentation: https://docs.mongodb.org/manual/tutorial/create-tailable-cursor/ https://docs.mongodb.org/manual/tutorial/create-tailable-cur... *Edit: clarified that the oplog is a capped collection for context
- timkham 11y agothanks for the clarification =)
- ninjakeyboard 11y ago*Akka Streams with an S
- nodesocket 11y agoI actually just wrote a node.js package that streams our MongoDB oplog to RethinKDB. It works quite well and based on the following NPM package https://github.com/cayasso/mongo-oplog https://github.com/cayasso/mongo-oplog. The oplog is super useful for auditing and eventing.
- einrealist 11y agoSome years, I wrote something similar: https://github.com/otto-de/mongodb-profiler https://github.com/otto-de/mongodb-profiler.
- johnymontana 11y agoThere is a nice higher-level Python wrapper around the oplog that is intended for syncing updates in MongoDB to other datastores [1]. We used this to enable streaming updates in MongoDB to Neo4j.[2] [1] https://github.com/mongodb-labs/mongo-connector https://github.com/mongodb-labs/mongo-connector [2] https://github.com/neo4j-contrib/neo4j_doc_manager https://github.com/neo4j-contrib/neo4j_doc_manager
- techdragon 11y agoThis is such a complex way to do what's built into RethinkDB. I just love how useful the activity streaming is.