6 ms·

How Raygun increased throughput 20x with .NET Core over Node.js

105 points by aliostad 9y ago

scandox 9y ago
> when we started to look at .NET Core in early 2016, it became quite obvious that being able to asynchronously hand off to our queuing service greatly improved throughput. Unfortunately, at the time, Node.js didn’t provide an easy mechanism to do this, while .NET Core had great concurrency capabilities from day one. This is a mildly maddening article - not because I have any emotional attachment to either platform - but because it never really gets to the heart of the matter. It's like they can tell us, but they'd have to kill us afterwards.
- andrewstuart 9y ago
  I'm with ya. All hype, no details. A 20X improvement cannot be attributed to .NET core, it must attributed substantially to some implementation/algorithmic issue that they resolved once they got to the .NET core platform. There's a big problem somewhere with node.js if it is 20X slower. If true, then its simply a bug or series of bugs to be fixed in node.js.
  - UK-AL 9y ago
    Just a basic request/response with barely anything thing in it, shows the .NET core is faster than node. https://stackoverflow.com/questions/43920942/unexpected-outcome-of-node-js-vs-asp-net-core-performance-test https://stackoverflow.com/questions/43920942/unexpected-outc... Obviously this isn't very scientific without a benchmark suite.
    - jamra 9y ago
      It's not 30 times faster. Also, it doesn't matter which one is faster unless there is. Huge speed improvement. Your web server is not the bottleneck.
      - UK-AL 9y ago
        "Your web server is not the bottleneck." - Highly depends on your setup. But removing any 3rd party libraries, and external requests. Generally .NET core is faster.
        SideburnsOfDoom 9y ago
        > But removing any 3rd party libraries, and external requests. Generally .NET core is faster. Is that with reference to "faster than .Net regular" or "faster than node.js"? You're right either way but I'm not sure that "it's just faster" is all that relevant. The "Just a basic request/response with barely anything thing in it" case removes anything of interest. For any normal non-trivial, non-demo setup, the web server is fine. I would look at the performance of data stores and other backends. The performance of "asynchronously hand off to our queuing service" is most likely the overriding factor in a 20 or 30x increase.
        UK-AL 9y ago
        Some have already optimised the heck out of their data retrieval and writes.
        SideburnsOfDoom 9y ago
        > Some have already optimised the heck out of their data retrieval and writes Then they have likely not very much to gain from framework speedup either. The fact that raygun could introduce a "asynchronously hand off to our queuing service" says quite clearly that "already optimised data retrieval and writes" is not the case that we are talking about.
        UK-AL 9y ago
        Asynchronously dropping something on to a queue, possibly leaves the queue client implementation as the suspect slowing node down. But assuming that dropping something on to a queue is incredibly fast. Switching how you serve your http requests can increase performance by some multiples because it would be the largest constant in time taken assuming a very fast queue drop. Although I agree, if your at that point your probably serving all your requests fine anyway.
  - emodendroket 9y ago
    It may also be that it's easier to write in performance issues or bugs unintentionally with Node.
- bleuarff 9y ago
  I agree with you. A bold claim with very vague explanations makes it a bit hard to believe. I'd love to have more details on how they pulled that off.
  - aliostad 9y ago
    this is yet the "more detailed" post. Here is the original story https://customers.microsoft.com/en-US/story/raygun https://customers.microsoft.com/en-US/story/raygun which I objected to https://twitter.com/aliostad/status/849186214045528064 https://twitter.com/aliostad/status/849186214045528064
vorpalhex 9y ago
The fact that they were only getting 1k reqs/sec with Node gives me concern. It clearly shows something went wrong there very early on at a very fundamental level. By no means is Node the end-all be-all for performance by any measure, but you should definitely be getting much much higher throughput than 1k reqs/sec. Simply booting up a single core http server should net you around 4-5k requests per second. Spin up an instance per core and you should be _at least_ in the 10k realm.
- coldtea 9y ago
  >Simply booting up a single core http server should net you around 4-5k requests per second. There's no such thing as "requests per second" generally. It's requests per second for a specific workload. So, whether Node can do 4-5k rps with "hello world" doesn't matter much. It's the same engine that needs to also do the further processing for each fuller request.
  - aliostad 9y ago
    My understanding is that they hand just off the payload to another queueing mechanism. So it just returns ACK and done.
- kompuuter 9y ago
  > The fact that they were only getting 1k reqs/sec with Node gives me concern. It clearly shows something went wrong there very early on at a very fundamental level. Because the program is performing CPU compute for each request. Node is single threaded. It's a shame that the Node project killed off the multithreaded web workers pull request. It sorely needs that functionality. Pools of node processes are a poor substitute.
  - allannienhuis 9y ago
    Care to elaborate on why additional node processes are a poor substitute? They are really easy to reason about, and to manage. What are the downsides? Not trolling, really interested in hearing about pros/cons.
endorphone 9y ago
This, and the AgeOfAscent piece it links to, both read like PR pieces commissioned by Microsoft. Both are absent of actual information to really determine what exactly they were testing, or what this means to anyone else's problem space. Because, of course, when you get to the fun of benchmarks there's always a faster options. With .NET Core offering them 20,000 requests per second (on a C3.large), that is a terribly low bar to hit. Again, maybe they're doing something amazing, but many frameworks have rates in the seven digits on that sort of hardware. And I know .NET Core can process basic requests in the six digits, so even it is hardly the limiting point, and it comes to the logic. https://www.techempower.com/benchmarks/# https://www.techempower.com/benchmarks/# Still terribly flawed, but better than someone rewriting an app and then gloating about speed improvements.
- SA500 9y ago
  It's an MS 'Customer Story' so it is a press release.
- revelation 9y ago
  Duh, yes, of course the logic makes it much slower than copying a L1 cached static response to the client. But then that is not a use case anyone cares very much about. The AgeOfAscent story is not a PR piece, the writer (Ben Adams) is a very active contributor to the Kestrel server behind .NET Core. If he's writing about improved performance it's probably because he wrote >1/3rd of the patches.
  - endorphone 9y ago
    Duh, yes, of course What a weirdly trite response given that you're essentially repeating what I said. And without specifics the linked piece, and its claims about benchmarks, is utterly meaningless. "Rewrote inefficient code. Now it's faster. Story at 11!" The AgeOfAscent story is not a PR piece Humorously it was likely a "copy a L1 cached static response" type benchmark. Okay, so it wasn't PR, it was self-aggrandizement (which is effectively PR). Got it. Though in this article it was linked as a performance improvement of "switching" to .NET Core, when really it was a story of terribly inefficient .NET Core code becoming better, though again in a nutshell it is meaningless. Cool.
ruffrey 9y ago
There is no technical information in this article. As a result it reads like Microsoft marketing and is off-putting to .NET Core, something I'd otherwise be interested in.
moomin 9y ago
I'm a .NET fan, but it sounds strongly like these guys knew what they were doing on .NET and not so much on node. It's amazing how many of these rewrite stories actually boil down to "and the new version didn't do some incredibly dumb things the old one did". If you really wanted throughput and large numbers of simultaneous connections, you'd be looking at Erlang, anyway.
xutopia 9y ago
I never trust any benchmark stories from Microsoft ever.
- cooldude67 9y ago
  Well there are plenty of public not run by Microsoft benchmarks that show this same trend: https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r14&hw=...
JTenerife 9y ago
Obviously they did something terribly wrong in the first place and now are showing off being all right. Netty would probably faster. As there're no details at all, I feel free to just guess :-). I recently found Node much faster then .NET Core for a very specific scenario: Elasticsearch - Node / ASP.NET Core (Kestrel) - NGINX - Client where every connection must be TLS. So, Node / ASP.NET Core have to decrypt traffic from Elasticsearch and encrypt to NGINX. For a minimal workload the whole trip took 50 ms with .NET Core and 20 ms with Node. Obviously there's something wrong with the .NET setup - maybe some setting with Kestrel and TLS. Anyway, that's my anecdote. And I wouldn't dare writing a block post how node is 2,5 times faster than .NET.
zbjornson 9y ago
It looks like they ran their last node.js benchmark around Sep 2015. I'm curious if this was still with node.js v0.x, or if they had already adopted the latest version of io.js (3 or maybe 4).
- drew-y 9y ago
  Or node.js v6.x for that matter. Seeing as io.js has merged back into node.
draw_down 9y ago
> From the questions we received around the specifics of our performance improvements, there seems to be two schools of thought: > 1. Of course it’s faster, Node is slow > 2. You must be doing Node wrong, it can be fast Allow me to offer a third :) When you rewrite something, it had better be faster! You know a lot more about how it works and how it's used than you knew at the outset. There was a recent post, about going back to Ruby after creating the first version in Clojure, that touches on this point.
- GoodbyeEarl 9y ago
  Hey! Where could I find this post?
  - draw_down 9y ago
    https://news.ycombinator.com/item?id=14405132 https://news.ycombinator.com/item?id=14405132
bluedino 9y ago
>> Node is a productive environment, and has a huge ecosystem around it, but frankly it hasn’t been designed for performance. Isn't/wasn't part of the hype behind node that it's fast? Compared to what?
markrendle 9y ago
Here's what I read: they had a solution written in Node.js; they rewrote the solution in .NET Core; it was 20x faster. It's anecdotal, sure, and Microsoft prompted them to write it, and YMMV, but they did a thing and it had a remarkable result so they've remarked on it. It's not realistic to expect them to take the time to build a representative sample that doesn't expose any of their proprietary business logic; they're running a business and have more important things to do. Anecdote of my own: I was working on a web bug that had to generate a few v4 UUIDs on every request, and using a version of Node for which the libuuid wrapper wasn't working, so I was using the fastest script-based generator I could find, but it was still too slow. A .NET Core version of the same code handled something like 40x the number of requests on the same hardware. If nothing else, it demonstrates that using the same solution for all your different problems is A Bad Thing, because there are surely things that .NET Core is not particularly good at either.
- jrs95 9y ago
  The biggest problem with .NET Core imo is that .NET just seems to be permanently uncool, despite how good it is. Maybe one day this industry will be less fashion driven...
  - wtf_is_up 9y ago
    I'd love to use it, but being Windows only for years was a non-starter for me.
    - StriverGuy 9y ago
      .NET core is cross platform. It is also far more performant than traditional .NET. The new framework is a huge leap forward.
      - mobilemidget 9y ago
        C / C++ is too, frameworks are for the lazy
garganzol 9y ago
The article is an excellent example of sneaky PR practices. Not only it lacks the technical details, it deceives the masses with insane amounts of artificial hype. "How AstroTurf invented astroturfing" would be a better title for it.
rikkus 9y ago
Has anyone any code they can show which performs the same job on both and is close to comparing apples to apples? It'd be interesting to me, not for performance comparison, but as a real world application structure comparison, a kind of more advanced Rosetta Code[1] example. [1] http://rosettacode.org/wiki/Rosetta_Code http://rosettacode.org/wiki/Rosetta_Code

105 points by aliostad 9y ago

> when we started to look at .NET Core in early 2016, it became quite obvious that being able to asynchronously hand off to our queuing service greatly improved throughput. Unfortunately, at the time, Node.js didn’t provide an easy mechanism to do this, while .NET Core had great concurrency capabilities from day one. This is a mildly maddening article - not because I have any emotional attachment to either platform - but because it never really gets to the heart of the matter. It's like they can tell us, but they'd have to kill us afterwards.

andrewstuart 9y ago
I'm with ya. All hype, no details. A 20X improvement cannot be attributed to .NET core, it must attributed substantially to some implementation/algorithmic issue that they resolved once they got to the .NET core platform. There's a big problem somewhere with node.js if it is 20X slower. If true, then its simply a bug or series of bugs to be fixed in node.js.
- UK-AL 9y ago
  Just a basic request/response with barely anything thing in it, shows the .NET core is faster than node. https://stackoverflow.com/questions/43920942/unexpected-outcome-of-node-js-vs-asp-net-core-performance-test https://stackoverflow.com/questions/43920942/unexpected-outc... Obviously this isn't very scientific without a benchmark suite.
  - jamra 9y ago
    It's not 30 times faster. Also, it doesn't matter which one is faster unless there is. Huge speed improvement. Your web server is not the bottleneck.
    - UK-AL 9y ago
      "Your web server is not the bottleneck." - Highly depends on your setup. But removing any 3rd party libraries, and external requests. Generally .NET core is faster.
      - SideburnsOfDoom 9y ago
        > But removing any 3rd party libraries, and external requests. Generally .NET core is faster. Is that with reference to "faster than .Net regular" or "faster than node.js"? You're right either way but I'm not sure that "it's just faster" is all that relevant. The "Just a basic request/response with barely anything thing in it" case removes anything of interest. For any normal non-trivial, non-demo setup, the web server is fine. I would look at the performance of data stores and other backends. The performance of "asynchronously hand off to our queuing service" is most likely the overriding factor in a 20 or 30x increase.
        UK-AL 9y ago
        Some have already optimised the heck out of their data retrieval and writes.
        SideburnsOfDoom 9y ago
        > Some have already optimised the heck out of their data retrieval and writes Then they have likely not very much to gain from framework speedup either. The fact that raygun could introduce a "asynchronously hand off to our queuing service" says quite clearly that "already optimised data retrieval and writes" is not the case that we are talking about.
        UK-AL 9y ago
        Asynchronously dropping something on to a queue, possibly leaves the queue client implementation as the suspect slowing node down. But assuming that dropping something on to a queue is incredibly fast. Switching how you serve your http requests can increase performance by some multiples because it would be the largest constant in time taken assuming a very fast queue drop. Although I agree, if your at that point your probably serving all your requests fine anyway.
- emodendroket 9y ago
  It may also be that it's easier to write in performance issues or bugs unintentionally with Node.
bleuarff 9y ago
I agree with you. A bold claim with very vague explanations makes it a bit hard to believe. I'd love to have more details on how they pulled that off.
- aliostad 9y ago
  this is yet the "more detailed" post. Here is the original story https://customers.microsoft.com/en-US/story/raygun https://customers.microsoft.com/en-US/story/raygun which I objected to https://twitter.com/aliostad/status/849186214045528064 https://twitter.com/aliostad/status/849186214045528064

vorpalhex 9y ago

The fact that they were only getting 1k reqs/sec with Node gives me concern. It clearly shows something went wrong there very early on at a very fundamental level. By no means is Node the end-all be-all for performance by any measure, but you should definitely be getting much much higher throughput than 1k reqs/sec. Simply booting up a single core http server should net you around 4-5k requests per second. Spin up an instance per core and you should be _at least_ in the 10k realm.

coldtea 9y ago
>Simply booting up a single core http server should net you around 4-5k requests per second. There's no such thing as "requests per second" generally. It's requests per second for a specific workload. So, whether Node can do 4-5k rps with "hello world" doesn't matter much. It's the same engine that needs to also do the further processing for each fuller request.
- aliostad 9y ago
  My understanding is that they hand just off the payload to another queueing mechanism. So it just returns ACK and done.
kompuuter 9y ago
> The fact that they were only getting 1k reqs/sec with Node gives me concern. It clearly shows something went wrong there very early on at a very fundamental level. Because the program is performing CPU compute for each request. Node is single threaded. It's a shame that the Node project killed off the multithreaded web workers pull request. It sorely needs that functionality. Pools of node processes are a poor substitute.
- allannienhuis 9y ago
  Care to elaborate on why additional node processes are a poor substitute? They are really easy to reason about, and to manage. What are the downsides? Not trolling, really interested in hearing about pros/cons.

endorphone 9y ago

This, and the AgeOfAscent piece it links to, both read like PR pieces commissioned by Microsoft. Both are absent of actual information to really determine what exactly they were testing, or what this means to anyone else's problem space. Because, of course, when you get to the fun of benchmarks there's always a faster options. With .NET Core offering them 20,000 requests per second (on a C3.large), that is a terribly low bar to hit. Again, maybe they're doing something amazing, but many frameworks have rates in the seven digits on that sort of hardware. And I know .NET Core can process basic requests in the six digits, so even it is hardly the limiting point, and it comes to the logic. https://www.techempower.com/benchmarks/# https://www.techempower.com/benchmarks/# Still terribly flawed, but better than someone rewriting an app and then gloating about speed improvements.

SA500 9y ago
It's an MS 'Customer Story' so it is a press release.
revelation 9y ago
Duh, yes, of course the logic makes it much slower than copying a L1 cached static response to the client. But then that is not a use case anyone cares very much about. The AgeOfAscent story is not a PR piece, the writer (Ben Adams) is a very active contributor to the Kestrel server behind .NET Core. If he's writing about improved performance it's probably because he wrote >1/3rd of the patches.
- endorphone 9y ago
  Duh, yes, of course What a weirdly trite response given that you're essentially repeating what I said. And without specifics the linked piece, and its claims about benchmarks, is utterly meaningless. "Rewrote inefficient code. Now it's faster. Story at 11!" The AgeOfAscent story is not a PR piece Humorously it was likely a "copy a L1 cached static response" type benchmark. Okay, so it wasn't PR, it was self-aggrandizement (which is effectively PR). Got it. Though in this article it was linked as a performance improvement of "switching" to .NET Core, when really it was a story of terribly inefficient .NET Core code becoming better, though again in a nutshell it is meaningless. Cool.

ruffrey 9y ago

There is no technical information in this article. As a result it reads like Microsoft marketing and is off-putting to .NET Core, something I'd otherwise be interested in.

moomin 9y ago

I'm a .NET fan, but it sounds strongly like these guys knew what they were doing on .NET and not so much on node. It's amazing how many of these rewrite stories actually boil down to "and the new version didn't do some incredibly dumb things the old one did". If you really wanted throughput and large numbers of simultaneous connections, you'd be looking at Erlang, anyway.

xutopia 9y ago

I never trust any benchmark stories from Microsoft ever.

cooldude67 9y ago
Well there are plenty of public not run by Microsoft benchmarks that show this same trend: https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=data-r14&hw=...

JTenerife 9y ago

Obviously they did something terribly wrong in the first place and now are showing off being all right. Netty would probably faster. As there're no details at all, I feel free to just guess :-). I recently found Node much faster then .NET Core for a very specific scenario: Elasticsearch - Node / ASP.NET Core (Kestrel) - NGINX - Client where every connection must be TLS. So, Node / ASP.NET Core have to decrypt traffic from Elasticsearch and encrypt to NGINX. For a minimal workload the whole trip took 50 ms with .NET Core and 20 ms with Node. Obviously there's something wrong with the .NET setup - maybe some setting with Kestrel and TLS. Anyway, that's my anecdote. And I wouldn't dare writing a block post how node is 2,5 times faster than .NET.

zbjornson 9y ago

It looks like they ran their last node.js benchmark around Sep 2015. I'm curious if this was still with node.js v0.x, or if they had already adopted the latest version of io.js (3 or maybe 4).

drew-y 9y ago
Or node.js v6.x for that matter. Seeing as io.js has merged back into node.

draw_down 9y ago

> From the questions we received around the specifics of our performance improvements, there seems to be two schools of thought: > 1. Of course it’s faster, Node is slow > 2. You must be doing Node wrong, it can be fast Allow me to offer a third :) When you rewrite something, it had better be faster! You know a lot more about how it works and how it's used than you knew at the outset. There was a recent post, about going back to Ruby after creating the first version in Clojure, that touches on this point.

GoodbyeEarl 9y ago
Hey! Where could I find this post?
- draw_down 9y ago
  https://news.ycombinator.com/item?id=14405132 https://news.ycombinator.com/item?id=14405132

bluedino 9y ago

>> Node is a productive environment, and has a huge ecosystem around it, but frankly it hasn’t been designed for performance. Isn't/wasn't part of the hype behind node that it's fast? Compared to what?

markrendle 9y ago

Here's what I read: they had a solution written in Node.js; they rewrote the solution in .NET Core; it was 20x faster. It's anecdotal, sure, and Microsoft prompted them to write it, and YMMV, but they did a thing and it had a remarkable result so they've remarked on it. It's not realistic to expect them to take the time to build a representative sample that doesn't expose any of their proprietary business logic; they're running a business and have more important things to do. Anecdote of my own: I was working on a web bug that had to generate a few v4 UUIDs on every request, and using a version of Node for which the libuuid wrapper wasn't working, so I was using the fastest script-based generator I could find, but it was still too slow. A .NET Core version of the same code handled something like 40x the number of requests on the same hardware. If nothing else, it demonstrates that using the same solution for all your different problems is A Bad Thing, because there are surely things that .NET Core is not particularly good at either.

jrs95 9y ago
The biggest problem with .NET Core imo is that .NET just seems to be permanently uncool, despite how good it is. Maybe one day this industry will be less fashion driven...
- wtf_is_up 9y ago
  I'd love to use it, but being Windows only for years was a non-starter for me.
  - StriverGuy 9y ago
    .NET core is cross platform. It is also far more performant than traditional .NET. The new framework is a huge leap forward.
    - mobilemidget 9y ago
      C / C++ is too, frameworks are for the lazy

garganzol 9y ago

The article is an excellent example of sneaky PR practices. Not only it lacks the technical details, it deceives the masses with insane amounts of artificial hype. "How AstroTurf invented astroturfing" would be a better title for it.

rikkus 9y ago

Has anyone any code they can show which performs the same job on both and is close to comparing apples to apples? It'd be interesting to me, not for performance comparison, but as a real world application structure comparison, a kind of more advanced Rosetta Code[1] example. [1] http://rosettacode.org/wiki/Rosetta_Code http://rosettacode.org/wiki/Rosetta_Code