3 ms·
If you put this behind, say a REST API vs using MongoDB directly, the clients could use long polling. If you look at Twitter's Kestrel, it was implemented with
by Todd 15y ago
If you put this behind, say a REST API vs using MongoDB directly, the clients could use long polling.
If you look at Twitter's Kestrel, it was implemented with a similar mechanism, albeit leveraging some capabilities that were intrinsic to memcached.
I built a little queuing prototype using long polling and it works very well. If you build it using an async server, you can run many clients while only using a few threads.
There's still the question of how to block and/or wake on an enqueue event to MongoDB vs. polling. The best approach may be to implement queueing using your REST API, leveraging all of the advantages of the async server and use MongoDB for a backing store or journal. This is similar to how Kestrel does it.
My prototype was very responsive, although I only got about 500 messages per second of throughput on my MBA.
- BenjaminCoe 15y agoI see what you're getting at. Behind the scenes, I am ultimately using an event loop :) Having said that, since MongoDB is a persistent connection, does long pulling over HTTP make sense for the client? I was thinking of doing a proof of concept in Node, that would at least behave like it's getting pushed messages rather than pulling ... But, I couldn't think of a way other than repeated setTimeouts. Any thoughts?
- Todd 15y agoNode would work well since it's designed to support async patterns. You're right that the challenge is the polling model that you have to implement when using MongoDB as the queue. To get throughput, you may have to use it as a journal and implement the queue within your process. Check out Kestrel on GitHub for an example.