5 ms·

I would be really curious to know what the overhead of actually creating the connection and loading the collections on each request would be. Is this anything m

I would be really curious to know what the overhead of actually creating the connection and loading the collections on each request would be. Is this anything more than a micro-optimization?

christkv 14y ago
Creating a new connection is slow compared to using an existing connection and also puts non-needed stress on the server as it needs to allocate a new thread for the socket. You'll also pretty quickly run out of file descriptors on your app server if you get hit by a little bit of load.
joshguthrie 14y ago
Coming from WAMP background, I had the same question when I first tried MongoDb databases with Node.JS and the consensus I read online back then (so, 3 years ago) was "Just connect once and don't bother anymore". Though the overhead can be rather minimal, it still exists: - Connect to the database - Query the Collection - Make a request - Close the connection Also, these four separate request are asynchronous, which means you could end up with X web requests having connected to the DB but not having a collection, Y requests having a collection and N requests waiting to be closed. This is not a real problem (as long as you don't have 4 servers connecting to the same Db with 65 thousand connections each :-) ) but having multiple CONCURRENT connections to the same server is pretty useless in a SINGLE-THREADED context like a Node.js app. Now, if you were to use PHP, that's a different matter...
- andypants 14y ago
  Actually, it can help in a node.js app if a single db connection can only complete one request at a time. The javascript in node.js executes in a single thread but a database connection is I/O. This is exactly what node.js is good at. With that said, you shouldn't create a connection per request, you should use a connection pool.
  - joshguthrie 14y ago
    Look at christkv comment for "why this is not a good idea in node.js". You may be thinking you can fire up multiple request at the same time (ie: fire the requests, let the I/O do the job, get all your requests back) but since you are in a single-threaded environment, you do exactly the same as I/O on a single connection (plus the overhead of opening new sockets and establishing hands shaking with the server).
    - andypants 14y ago
      christkv's comment is about creating a new connection per request. As I said, I agree, you should use a connection pool instead. When the web request comes in, you already have multiple database connections waiting for you. The connection overhead is not an issue then. And yes, that is exactly what I am thinking. More database connections = more concurrent database requests. I don't understand your reasoning about being in a single threaded environment. You made a statement but you haven't explained it. From what I can see, this problem is simply serial vs parallel. Is there something I'm missing? I don't see how the fact that node.js is single threaded changes the situation since the database requests aren't working within the node.js context. Edit: sorry, my previous reply may have been confusing. I was responding to your comment that having multiple connections is useless in a single-threaded environment. I was not suggesting that it may be helpful for the OP to make a new connection for each incoming http request.
      - joshguthrie 14y ago
        Why not try to benchmark the results and see how it really goes? :-) In both cases, the same process of reading/writing to one or more connection sockets happen, you send and receive the same content to MongoDb: you can not send twice as much queries, you're gonna send a query, THEN you're gonna send another, then you get an answer THEN you get another answer. Everything happens in the same sequence. Also take in account MongoDb's own concurrency properties.
        andypants 14y ago

I would be really curious to know what the overhead of actually creating the connection and loading the collections on each request would be. Is this anything more than a micro-optimization?

christkv 14y ago

Creating a new connection is slow compared to using an existing connection and also puts non-needed stress on the server as it needs to allocate a new thread for the socket. You'll also pretty quickly run out of file descriptors on your app server if you get hit by a little bit of load.

joshguthrie 14y ago

Coming from WAMP background, I had the same question when I first tried MongoDb databases with Node.JS and the consensus I read online back then (so, 3 years ago) was "Just connect once and don't bother anymore". Though the overhead can be rather minimal, it still exists: - Connect to the database - Query the Collection - Make a request - Close the connection Also, these four separate request are asynchronous, which means you could end up with X web requests having connected to the DB but not having a collection, Y requests having a collection and N requests waiting to be closed. This is not a real problem (as long as you don't have 4 servers connecting to the same Db with 65 thousand connections each :-) ) but having multiple CONCURRENT connections to the same server is pretty useless in a SINGLE-THREADED context like a Node.js app. Now, if you were to use PHP, that's a different matter...

andypants 14y ago
Actually, it can help in a node.js app if a single db connection can only complete one request at a time. The javascript in node.js executes in a single thread but a database connection is I/O. This is exactly what node.js is good at. With that said, you shouldn't create a connection per request, you should use a connection pool.
- joshguthrie 14y ago
  Look at christkv comment for "why this is not a good idea in node.js". You may be thinking you can fire up multiple request at the same time (ie: fire the requests, let the I/O do the job, get all your requests back) but since you are in a single-threaded environment, you do exactly the same as I/O on a single connection (plus the overhead of opening new sockets and establishing hands shaking with the server).
  - andypants 14y ago
    christkv's comment is about creating a new connection per request. As I said, I agree, you should use a connection pool instead. When the web request comes in, you already have multiple database connections waiting for you. The connection overhead is not an issue then. And yes, that is exactly what I am thinking. More database connections = more concurrent database requests. I don't understand your reasoning about being in a single threaded environment. You made a statement but you haven't explained it. From what I can see, this problem is simply serial vs parallel. Is there something I'm missing? I don't see how the fact that node.js is single threaded changes the situation since the database requests aren't working within the node.js context. Edit: sorry, my previous reply may have been confusing. I was responding to your comment that having multiple connections is useless in a single-threaded environment. I was not suggesting that it may be helpful for the OP to make a new connection for each incoming http request.
    - joshguthrie 14y ago
      Why not try to benchmark the results and see how it really goes? :-) In both cases, the same process of reading/writing to one or more connection sockets happen, you send and receive the same content to MongoDb: you can not send twice as much queries, you're gonna send a query, THEN you're gonna send another, then you get an answer THEN you get another answer. Everything happens in the same sequence. Also take in account MongoDb's own concurrency properties.
      - andypants 14y ago