3 ms·
When I need to process sometimes millions of documents from a production mongoDB instance I normally push all the objectID into rabbitMQ then spin up N workers
by vorticalbox 3y ago
When I need to process sometimes millions of documents from a production mongoDB instance I normally push all the objectID into rabbitMQ then spin up N workers that each fetch a single message at a time.
- rdedev 3y agoIs there a reason why you use rabbitmq instead of just using the whats available in the language (I understand if those primitives are hard to work with)
- tempay 3y agoI'd guess they're using multiple machines.
- vorticalbox 3y agoAs said people using multiple ec2s to process the data but the biggest reason is so I don't have to run it all in one go. Rabbit is used by a few of our services so is always up, means I can run for a few hours then come back to it another day. Also error handling, if one fails I just don't ack the message, after everything is processed I only have failed ones. Can slowly process the messages to see what is wrong
- samsquire 3y agoThanks for sharing, I think this topic is very interesting. I played about with Kafka consumer groups to get the same effect.
- paulddraper 3y ago> then spin up N workers And how do you do that? Because...that is also what the multiprocessing module does.