4 ms·

With this kind of architecture, do you risk reaching a feedback loop where too many queries overload the DB and slow it down (especially given how MongoDB deleg

With this kind of architecture, do you risk reaching a feedback loop where too many queries overload the DB and slow it down (especially given how MongoDB delegates caching to the OS), causing more messages to accumulate, increasing pressure on both the application server & the DB, leading to a full-on thrashy death spiral? Or is this mitigated somehow? Basically, what's the failure mode when usage spikes more than anticipated?

tmeasday 13y ago
It's definitely a concern, but I think it's mitigated somewhat by the oplog-tailing approach. Writes go straight into Mongo, as usual, it's just reads from the oplog that potentially tie up resources on the webserver. Those reads are throttled, so I think the net result of too many writes is just a choked up webserver, and slow "realtime". Potentially a future architecture moves the oplog processor onto it's own server and the effect is isolated to the realtime part.
- panarky 13y ago
  So with oplog tailing, I can scale by adding multiple Meteor processes behind a reverse proxy with sticky sessions. It seems that since oplog tailing doesn't work with a sharded database, then each additional Meteor process must also tail the entire oplog and evaluate all transactions for all Meteor processes. How does this work for a site with, say, 100 transactions per second and a dozen subscriptions to observe? Does it saturate the Node event loop, or are there plenty of ticks left for the application? And if I use 'skip' and 'limit' to page through large result sets, it looks like oplog tailing isn't an option. Is there a better way to keep the client data small and take advantage of oplog tailing?
  - imslavko 13y ago
    You can also paginate by `$gt` and `$lt` operators. For example, if you have a list of posts, they are probably displayed in some order on the screen. Most likely they are ordered by date. You can maintain the pagination by presenting all posts older than X but newer than Y.

tmeasday 13y ago

It's definitely a concern, but I think it's mitigated somewhat by the oplog-tailing approach. Writes go straight into Mongo, as usual, it's just reads from the oplog that potentially tie up resources on the webserver. Those reads are throttled, so I think the net result of too many writes is just a choked up webserver, and slow "realtime". Potentially a future architecture moves the oplog processor onto it's own server and the effect is isolated to the realtime part.

panarky 13y ago
So with oplog tailing, I can scale by adding multiple Meteor processes behind a reverse proxy with sticky sessions. It seems that since oplog tailing doesn't work with a sharded database, then each additional Meteor process must also tail the entire oplog and evaluate all transactions for all Meteor processes. How does this work for a site with, say, 100 transactions per second and a dozen subscriptions to observe? Does it saturate the Node event loop, or are there plenty of ticks left for the application? And if I use 'skip' and 'limit' to page through large result sets, it looks like oplog tailing isn't an option. Is there a better way to keep the client data small and take advantage of oplog tailing?
- imslavko 13y ago
  You can also paginate by `$gt` and `$lt` operators. For example, if you have a list of posts, they are probably displayed in some order on the screen. Most likely they are ordered by date. You can maintain the pagination by presenting all posts older than X but newer than Y.