4 ms·

I implemented the Ringo app for this benchmark and of course ran it against Node and a couple of others to see how we would perform in this neighborhood before

I implemented the Ringo app for this benchmark and of course ran it against Node and a couple of others to see how we would perform in this neighborhood before I opened the pull request. And since that day I've been wondering: why does NodeJs (=V8 JS engine in C) talking to MongoDB have higher response times and latency than Ringo (=Rhino JS engine on JVM) talking to MySQL. The only thing where Node beats us JVM guys seems to be the JSON response test.

ricardobeat 13y ago
Node favours concurrency over raw speed; calls deferred with process.nextTick and callbacks end up costing time, in exchange for better concurrency. I think a blocking driver could leave Ringo in the dust, but it would be useless.
- oberhamsi 13y ago
  I will test that. Should the latency stay more constant with higher concurrency? Or what am I searching for? Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32, 64, 128, 256): nodejs (mongodb raw) 12,541 22,073 26,761 26,461 28,316 28,856 ringojs (mysql raw) 13,190 23,556 27,758 30,697 31,382 31,997 both look like they got room to grow
  - rdtsc 13y ago
    I think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.
    - bhauer 13y ago
      Hi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web server's socket listener thread) needs to maintain a small queue of requests to hand off to the application server's worker threads. It doesn't make the server any faster at completing those requests. I've written some more about this at my personal blog [2]. [1] Caveat: Some frameworks appear to have locking or resource contention issues and do not saturate the CPU cores. We will attempt to capture CPU utilization stats in future rounds since this might be of interest to readers and framework maintainers. But increasing concurrency would not increase CPU utilization in these scenarios either. [2] http://tiamat.tsotech.com/rps-vs-connections http://tiamat.tsotech.com/rps-vs-connections
      - ricardobeat 13y ago
        The point is benchmarking concurrency limits, not that it would improve raw performance.
        bhauer 13y ago
        Ah, okay. That's interesting, I suppose. You basically are interested in seeing at what concurrency level the server starts spitting back 500-series responses (or simply doesn't provide a response). Basically, how many concurrent requests are needed before the server's inbound request queue overflows. We could test that out at some point.
        oberhamsi 13y ago
        ah! I didn't think of it like that. That would be interesting to see... at which concurrency level the environments crap out.
      - rdtsc 13y ago
        Hi bhauer, > concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Well websocket connections are becoming more and more popular. Maybe that's a different benchmark. But the level of concurrency is pretty important. It basically tells the story of what happens to a "slashdotted" server. If nothing crazy like that happens than most servers might be ok, just maybe have a little higher latency. It is when shit hits the fan that different servers start separating from the herd. Some gracefully slow down, some scale smoothly across CPUs, some start throwing socket errors. Who cares about these issues? Well anyone who becomes successful. If there are no visitors and no customers and only a GET request here and there every 10 minutes, then those places could really just use any server. A simple Perl or Ruby one will do. Now those that grow and see customers they will be interested in what happens in cases like that. There is a traffic spike at launch of new product so now there is a 200% increase in traffic for that one day and it tapers off. Maybe we just come from a different background and that's why they focus is on different metrics. > It doesn't make the server any faster at completing those requests. But I am not sure what story does benchmarking the servers at an artificial level of concurrency tells us. Maybe it helps those that have a throttling/balancing proxy that always sets the number of connection to 256 at most and otherwise balances out the rest to other servers... And I am not sure if a the heuristic that "If it can handle 2456 requests/second with a single connection at a time" can be extrapolated and implies then it can "handle 2456 concurrent connections in a single second".

ricardobeat 13y ago

Node favours concurrency over raw speed; calls deferred with process.nextTick and callbacks end up costing time, in exchange for better concurrency. I think a blocking driver could leave Ringo in the dust, but it would be useless.

oberhamsi 13y ago
I will test that. Should the latency stay more constant with higher concurrency? Or what am I searching for? Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32, 64, 128, 256): nodejs (mongodb raw) 12,541 22,073 26,761 26,461 28,316 28,856 ringojs (mysql raw) 13,190 23,556 27,758 30,697 31,382 31,997 both look like they got room to grow
- rdtsc 13y ago
  I think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.
  - bhauer 13y ago
    Hi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web server's socket listener thread) needs to maintain a small queue of requests to hand off to the application server's worker threads. It doesn't make the server any faster at completing those requests. I've written some more about this at my personal blog [2]. [1] Caveat: Some frameworks appear to have locking or resource contention issues and do not saturate the CPU cores. We will attempt to capture CPU utilization stats in future rounds since this might be of interest to readers and framework maintainers. But increasing concurrency would not increase CPU utilization in these scenarios either. [2] http://tiamat.tsotech.com/rps-vs-connections http://tiamat.tsotech.com/rps-vs-connections
    - ricardobeat 13y ago
      The point is benchmarking concurrency limits, not that it would improve raw performance.
      - bhauer 13y ago
        Ah, okay. That's interesting, I suppose. You basically are interested in seeing at what concurrency level the server starts spitting back 500-series responses (or simply doesn't provide a response). Basically, how many concurrent requests are needed before the server's inbound request queue overflows. We could test that out at some point.
      - oberhamsi 13y ago
        ah! I didn't think of it like that. That would be interesting to see... at which concurrency level the environments crap out.
    - rdtsc 13y ago
      Hi bhauer, > concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Well websocket connections are becoming more and more popular. Maybe that's a different benchmark. But the level of concurrency is pretty important. It basically tells the story of what happens to a "slashdotted" server. If nothing crazy like that happens than most servers might be ok, just maybe have a little higher latency. It is when shit hits the fan that different servers start separating from the herd. Some gracefully slow down, some scale smoothly across CPUs, some start throwing socket errors. Who cares about these issues? Well anyone who becomes successful. If there are no visitors and no customers and only a GET request here and there every 10 minutes, then those places could really just use any server. A simple Perl or Ruby one will do. Now those that grow and see customers they will be interested in what happens in cases like that. There is a traffic spike at launch of new product so now there is a 200% increase in traffic for that one day and it tapers off. Maybe we just come from a different background and that's why they focus is on different metrics. > It doesn't make the server any faster at completing those requests. But I am not sure what story does benchmarking the servers at an artificial level of concurrency tells us. Maybe it helps those that have a throttling/balancing proxy that always sets the number of connection to 256 at most and otherwise balances out the rest to other servers... And I am not sure if a the heuristic that "If it can handle 2456 requests/second with a single connection at a time" can be extrapolated and implies then it can "handle 2456 concurrent connections in a single second".