6 ms·

This kind of benchmark is completely useless. You tune the heck out of node.js and then take another tool without tuning it (JVM, apache, nginx etc), give it a

This kind of benchmark is completely useless. You tune the heck out of node.js and then take another tool without tuning it (JVM, apache, nginx etc), give it a ridiculous task that you'll never find in real world and present your results as if they are meaningful. Why do people still waste time doing it?

romanovcode 10y ago
Actually it's kind of funny how Node community are obsessed about benchmark and speed because Node is very slow on CPU and cannot even do parallelism properly.
- parhurs 10y ago
  Could you mention what this is based on, please?
- k__ 10y ago
  I remember the dev of uws got some hate last year. Ppl were complaining that uws would only perform well on C++ stacks and be slow on Node. So I guess he simply wanted to prove them wrong.
- anilgulecha 10y ago
  That's completely incorrect. Well known cluster modules allows very easy parallelism at any level in your application. Plus Node is among the fastest interpreted languages around today -- coming close to the jvm in performance.
  - sanjayts 10y ago
    I am awfully wary of these statements which paint languages like Python (via PyPy), Javascript (via Node) as very close competitors of the JVM. Once the JIT engine kicks in, on "real" workloads, JVM beats the lights out of these carefully tuned interpreted languages on a CPU intensive workload.
    - coldtea 10y ago
      >Once the JIT engine kicks in, on "real" workloads, JVM beats the lights out of these carefully tuned interpreted languages on a CPU intensive workload. For one, JS is also JITed. Second we have video players and other tasks done on native JS, which would be impossibly slow on say Python. Second, JS can also be compiled -- there's asm.js and WebAssembly coming down the road. So, yes, it might be slower than the JVM, but not that slower for most practical purposes.
      - sanjayts 10y ago
        > For one, JS is also JITed But not all JITs are equal; that's like putting Brainfuck in the mix because it has a JIT. It is worth noting that JVM JIT has years of research behind it and being statically typed only adds to he benefits. > So, yes, it might be slower than the JVM, but not that slower for most practical purposes. Sure, my point is that the "not that slower" varies on lot depending on the kind of computation would run and having a notion that these dynamic languages are fast enough just perpetuates the misunderstanding that there exists free lunch...
        coldtea 10y ago
        >It is worth noting that JVM JIT has years of research I hear that a lot and it's a moot point. It's not like the same research is not available to those doing the JS JITs. Unless we're talking about patents, techniques for faster JITing are widely known, and get propagated to newer languages and runtimes all the time. And in fact, even the people are usually the same (e.g. people that started the initial fast JITs in the days of Smalltalk, then went to JVM, and now work on V8). >Sure, my point is that the "not that slower" varies on lot depending on the kind of computation would run and having a notion that these dynamic languages are fast enough just perpetuates the misunderstanding that there exists free lunch... Well, certainly fast enough for web apps, where we have been using 10x slower languages with no JITs and huge overheads.
- roryrjb 10y ago
  IO-intensive (what this module would be used for) != CPU-intensive. When considering "pure" code in CPython, YARV or Perl, JavaScript easily out performs all of those, when thinking of dynamic languages which the Node community would be mostly comparing against. Obviously compiled and statically compiled languages are a different ball park in terms of both performance and culture.
  - Yokohiii 10y ago
    Well isn't that a reason to compare against real competitors instead of showing off against the ones you can beat?
    - roryrjb 10y ago
      Yes definitely, just really pointing out that the Node community probably has more in common to those languages community and motivations than the C++, Java, Haskell community (just to pick some examples) for many differing reasons.
- coldtea 10y ago
  >because Node is very slow on CPU and cannot even do parallelism properly Compared to what? Python? Ruby? PHP? None of which can do "parallelism properly" and all of which are from 2 times to an order of magnitude slower than v8 (Node's engine). Even Go can't do parallelism properly -- it's model is full of manual handling of deadlocks and races. Erlang, yes.
rubyfan 10y ago
Not totally useless. This shows the performance and overhead of the library/framework not the task. Many readers should have a feel for their own use cases and be able to relate them to "hello world" benchmark responses, for example I usually take it to mean divide stated performance by 10 immediately if a simple DB query is involved, etc. Also if today you are using one of the compared setups you should know what performance you currently have and what tuning went into it to get a relativity.
- Yokohiii 10y ago
  This is why they should benchmark against a raw c implementation with same feature set. People seem to try to raise the bar with just ignoring what is ahead of them.
  - rubyfan 10y ago
    If you hit the README for the project you'll see they've done comparisons against a number of their peers.
- Strom 10y ago
  Microbenchmark results don't linearly scale to everything else. Just because language X can print "Hello, World!" 2x faster than language Y doesn't mean that every other operation is also 2x faster. For example a huge factor is algorithm quality. Language X may be fast at "Hello, World!", but then proceed to have QuickSort as its standard sorting function, while language Y has Timsort. [1] Language X may have a nicely optimized C library for hashing, while language Y has AVX2 optimized ASM. Some languages don't even have a wide & well-optimized standard library. Thus you can only really tell how good a language/library is for your usecase if you test with an actual real world scenario. Additionally, the ultimate microbenchmark winning code is one that does every trick in the book while not caring about anything else. This means hooking the kernel, unloading every kernel module / driver that isn't necessary for the microbenchmark, and doing the microbenchmark work at ring0 with absolute minimum overhead. Written in ASM, which is implanted by C code, which is launched by node.js. Then, if there's any data dependant processing in the microbenchmark, the winning code will precompute everything and load the full 2 TB of precomputed data into RAM. The playing field is even, JVM & Apache, or whatever else is the competition will also be run on this 2 TB RAM machine of course. They just won't use it, because they aren't designed to deliver the best results in this single microbenchmark. The point is that, not only don't microbenchmark results mean linear scaling for other work, but the techniques to achieve the microbenchmark results may even be detrimental to everything else! -- [1] For some data sets QuickSort is actually faster. Goes to show you that the best choice is highly dependant on actual use.
sunilkumarc 10y ago
I don't understand why you would say this benchmark is useless. As we can see from the wiki, the core http module can handle just 65k requests/second. But the new websockets approach can handle a million requests/second. I think this is astonishing.
- Strom 10y ago
  First off it uses HTTP/1.1 pipelining. There's not a single actively developed browser in existance that supports HTTP/1.1 pipelining out of the box. [1] Thus you certainly can't use this for web development. Secondly, the post doesn't seem to mention this but I'm willing to bet that this microbenchmark, like all others like this, are doing all these million requests from a single client that's located on the same machine. How many real world use cases are there where a single localhost client will do a million requests per second and also supports HTTP/1.1 pipelining? -- [1] https://en.wikipedia.org/wiki/HTTP_pipelining#Implementation_in_web_browsers https://en.wikipedia.org/wiki/HTTP_pipelining#Implementation...
  - z3t4 10y ago
    basically all apps now a day communicate via http to avoid firewalls.
  - nitely 10y ago
    It's useless for other reasons. Pipelining and non-idempotent methods (i.E POST) don't go togheter.
- Yokohiii 10y ago
  Having lower latency and bigger throughput is always good. But will it have any impact in real apps? Memory management and IO aren't gone just because your http stack is fast. The average node app will probably fall way behind just because of GC.
  - coldtea 10y ago
    >Having lower latency and bigger throughput is always good. But will it have any impact in real apps? Obviously yes. >Memory management and IO aren't gone just because your http stack is fast. Obviously yes again. But they are helped by it.
Strom 10y ago
These not-real-world microbenchmarks are definitely useless from an engineering perspective. However they aren't completely useless. Their use is marketing. It gets the word out to developers that this product X is really good! So what if it doesn't translate to real world scenarios or even if the numbers are completely fabricated and you can't even reproduce this under lab conditions. [1] Very few people care enough to look at things that closely. Just seeing a bunch of posts claiming product X is really good is enough to leave a strong impression that product X really is that great. Perception is reality, and perception is usually better influenced by massive claims (even if untrue) rather than realstic iterative progress. Our product is 5% faster than state of the art! just doesn't have that viral headline nature that you need to win over the hearts of the masses. As for why would someone do this. Maybe they don't know better, or maybe they are doing it because they have decided to invest in some technology tribe and thus profit from that tribe surviving, and even more from growing. This is a pretty automatic behavior for humans. Take any tribal war, e.g. XBox One vs PS4. Those who happen to own a XBox One (perhaps as a gift) can be seen at various places passionately arguing that XBox One is better than PS4, even if objectively it has worse hardware and less highly acclaimed exlcusive games. The person is on the XBox One tribe, and working towards getting more users to own XBox One will mean that more developer investments are also made towards the XBox One thanks to the bigger userbase. Thus even if the original claims to get users into the tribe were false, if growth is big enough it may work out well enough at the end. -- [1] The RethinkDB postmortem [2] had a great paragraph about these microbenchmarks. People wanted RethinkDB to be fast on workloads they actually tried, rather than “real world” workloads we suggested. For example, they’d write quick scripts to measure how long it takes to insert ten thousand documents without ever reading them back. MongoDB mastered these workloads brilliantly, while we fought the losing battle of educating the market. [2] http://www.defstartup.org/2017/01/18/why-rethinkdb-failed.html http://www.defstartup.org/2017/01/18/why-rethinkdb-failed.ht...
- deleted 10y ago
  [deleted]
- mcherm 10y ago
  Your example from RethinkDB really struck home to me. The idea that superior technology might lose out due to poor marketing or (in this case) a system that is optimized for the real world rather than being optimized for benchmarks really disturbs me. And (this is just my personality) I don't like being disturbed about something without trying to "solve" it. So here's my best thought on how to handle the situation where a team feels that they have a superior product which is losing out to another product that is optimized for benchmarks: > Provide a setting called something like "speed mode". In this mode it is completely optimized for the benchmarks, at the cost of everything else. Default to running without "speed mode", but for anyone who is running benchmarks ask them if they've tried it in "speed mode". A truly competent evaluator will insist on trying the system with the options that are really used in the real world, but then the competent evaluator won't be using an unreliable benchmark anyway. Anyone running the benchmarks just to see how well it works will be likely to turn on something named "speed mode", or at least to do so if asked to. Forums will eventually fill up with people recommending "for real-world loads, you should disable 'speed mode' as it doesn't actually speed them up". Hmm... sounds cool, but I'm not so sure it would actually work. The danger is that you would instead develop a reputation for "cheating" on benchmarks. This is why I'm not very good at marketing.
coldtea 10y ago
>This kind of benchmark is completely useless. On the contrary, that's the only useful type of benchmark. I don't care for "load simulation" full benchmarks, with loads and usage patterns that will be invariably different than mine, and which tell me nothing much. Microbenchmarks on the other hand, are constrained to very specific situations (like the above query), and as such can be very precise in the numbers they give. I know that if I use a similar machine and Node version, and have the same query, I will get the same performance. And that's exactly what programmers use to identify pain points ("hmm, this kind of response handling is slow") and fix it. Isolated and targeted microbenchmarks. I don't care for a "full" benchmark to tell me that "things will slow down with business logic and DB queries". Well, DUH!
- xienze 10y ago
  > I don't care for a "full" benchmark to tell me that "things will slow down with business logic and DB queries". Well, DUH! Yes but you don't know exactly how much those things will differ between languages, which is important. For instance, you may say "wow, Node smokes Java in this echo server benchmark, I'm sold!", only to later find out that e.g., DB queries run 3x slower in Node than Java. Suddenly a more real-world benchmark makes sense...
  - coldtea 10y ago
    >only to later find out that e.g., DB queries run 3x slower in Node than Java. Suddenly a more real-world benchmark makes sense No, then I just need an additional DB query microbenchmark.
jondubois 10y ago
I think it shows that the author(s) behind the library is/are committed to improving performance. Sure, a lot of companies like to publish benchmarks like this to make their product look better than it is (E.g. only showing the good parts), but in this case I know the author and I can vouch that he is independent and uncompromising. You could argue that the baseline performance of a library doesn't matter as much once you start adding lots of custom logic on top, but it's still highly relevant for lightweight workloads (which are actually quite common E.g. Basic chat systems).
- enraged_camel 10y ago
  >>I think it shows that the author(s) behind the library is/are committed to improving performance. No, it shows that they care about benchmarks, which are rarely if ever representative of real-life scenarios.

romanovcode 10y ago

Actually it's kind of funny how Node community are obsessed about benchmark and speed because Node is very slow on CPU and cannot even do parallelism properly.

parhurs 10y ago
Could you mention what this is based on, please?
k__ 10y ago
I remember the dev of uws got some hate last year. Ppl were complaining that uws would only perform well on C++ stacks and be slow on Node. So I guess he simply wanted to prove them wrong.
anilgulecha 10y ago
That's completely incorrect. Well known cluster modules allows very easy parallelism at any level in your application. Plus Node is among the fastest interpreted languages around today -- coming close to the jvm in performance.
- sanjayts 10y ago
  I am awfully wary of these statements which paint languages like Python (via PyPy), Javascript (via Node) as very close competitors of the JVM. Once the JIT engine kicks in, on "real" workloads, JVM beats the lights out of these carefully tuned interpreted languages on a CPU intensive workload.
  - coldtea 10y ago
    >Once the JIT engine kicks in, on "real" workloads, JVM beats the lights out of these carefully tuned interpreted languages on a CPU intensive workload. For one, JS is also JITed. Second we have video players and other tasks done on native JS, which would be impossibly slow on say Python. Second, JS can also be compiled -- there's asm.js and WebAssembly coming down the road. So, yes, it might be slower than the JVM, but not that slower for most practical purposes.
    - sanjayts 10y ago
      > For one, JS is also JITed But not all JITs are equal; that's like putting Brainfuck in the mix because it has a JIT. It is worth noting that JVM JIT has years of research behind it and being statically typed only adds to he benefits. > So, yes, it might be slower than the JVM, but not that slower for most practical purposes. Sure, my point is that the "not that slower" varies on lot depending on the kind of computation would run and having a notion that these dynamic languages are fast enough just perpetuates the misunderstanding that there exists free lunch...
      - coldtea 10y ago
        >It is worth noting that JVM JIT has years of research I hear that a lot and it's a moot point. It's not like the same research is not available to those doing the JS JITs. Unless we're talking about patents, techniques for faster JITing are widely known, and get propagated to newer languages and runtimes all the time. And in fact, even the people are usually the same (e.g. people that started the initial fast JITs in the days of Smalltalk, then went to JVM, and now work on V8). >Sure, my point is that the "not that slower" varies on lot depending on the kind of computation would run and having a notion that these dynamic languages are fast enough just perpetuates the misunderstanding that there exists free lunch... Well, certainly fast enough for web apps, where we have been using 10x slower languages with no JITs and huge overheads.
roryrjb 10y ago
IO-intensive (what this module would be used for) != CPU-intensive. When considering "pure" code in CPython, YARV or Perl, JavaScript easily out performs all of those, when thinking of dynamic languages which the Node community would be mostly comparing against. Obviously compiled and statically compiled languages are a different ball park in terms of both performance and culture.
- Yokohiii 10y ago
  Well isn't that a reason to compare against real competitors instead of showing off against the ones you can beat?
  - roryrjb 10y ago
    Yes definitely, just really pointing out that the Node community probably has more in common to those languages community and motivations than the C++, Java, Haskell community (just to pick some examples) for many differing reasons.
coldtea 10y ago
>because Node is very slow on CPU and cannot even do parallelism properly Compared to what? Python? Ruby? PHP? None of which can do "parallelism properly" and all of which are from 2 times to an order of magnitude slower than v8 (Node's engine). Even Go can't do parallelism properly -- it's model is full of manual handling of deadlocks and races. Erlang, yes.

rubyfan 10y ago

Not totally useless. This shows the performance and overhead of the library/framework not the task. Many readers should have a feel for their own use cases and be able to relate them to "hello world" benchmark responses, for example I usually take it to mean divide stated performance by 10 immediately if a simple DB query is involved, etc. Also if today you are using one of the compared setups you should know what performance you currently have and what tuning went into it to get a relativity.

Yokohiii 10y ago
This is why they should benchmark against a raw c implementation with same feature set. People seem to try to raise the bar with just ignoring what is ahead of them.
- rubyfan 10y ago
  If you hit the README for the project you'll see they've done comparisons against a number of their peers.
Strom 10y ago
Microbenchmark results don't linearly scale to everything else. Just because language X can print "Hello, World!" 2x faster than language Y doesn't mean that every other operation is also 2x faster. For example a huge factor is algorithm quality. Language X may be fast at "Hello, World!", but then proceed to have QuickSort as its standard sorting function, while language Y has Timsort. [1] Language X may have a nicely optimized C library for hashing, while language Y has AVX2 optimized ASM. Some languages don't even have a wide & well-optimized standard library. Thus you can only really tell how good a language/library is for your usecase if you test with an actual real world scenario. Additionally, the ultimate microbenchmark winning code is one that does every trick in the book while not caring about anything else. This means hooking the kernel, unloading every kernel module / driver that isn't necessary for the microbenchmark, and doing the microbenchmark work at ring0 with absolute minimum overhead. Written in ASM, which is implanted by C code, which is launched by node.js. Then, if there's any data dependant processing in the microbenchmark, the winning code will precompute everything and load the full 2 TB of precomputed data into RAM. The playing field is even, JVM & Apache, or whatever else is the competition will also be run on this 2 TB RAM machine of course. They just won't use it, because they aren't designed to deliver the best results in this single microbenchmark. The point is that, not only don't microbenchmark results mean linear scaling for other work, but the techniques to achieve the microbenchmark results may even be detrimental to everything else! -- [1] For some data sets QuickSort is actually faster. Goes to show you that the best choice is highly dependant on actual use.

sunilkumarc 10y ago

I don't understand why you would say this benchmark is useless. As we can see from the wiki, the core http module can handle just 65k requests/second. But the new websockets approach can handle a million requests/second. I think this is astonishing.

Strom 10y ago
First off it uses HTTP/1.1 pipelining. There's not a single actively developed browser in existance that supports HTTP/1.1 pipelining out of the box. [1] Thus you certainly can't use this for web development. Secondly, the post doesn't seem to mention this but I'm willing to bet that this microbenchmark, like all others like this, are doing all these million requests from a single client that's located on the same machine. How many real world use cases are there where a single localhost client will do a million requests per second and also supports HTTP/1.1 pipelining? -- [1] https://en.wikipedia.org/wiki/HTTP_pipelining#Implementation_in_web_browsers https://en.wikipedia.org/wiki/HTTP_pipelining#Implementation...
- z3t4 10y ago
  basically all apps now a day communicate via http to avoid firewalls.
- nitely 10y ago
  It's useless for other reasons. Pipelining and non-idempotent methods (i.E POST) don't go togheter.
Yokohiii 10y ago
Having lower latency and bigger throughput is always good. But will it have any impact in real apps? Memory management and IO aren't gone just because your http stack is fast. The average node app will probably fall way behind just because of GC.
- coldtea 10y ago
  >Having lower latency and bigger throughput is always good. But will it have any impact in real apps? Obviously yes. >Memory management and IO aren't gone just because your http stack is fast. Obviously yes again. But they are helped by it.

Strom 10y ago

These not-real-world microbenchmarks are definitely useless from an engineering perspective. However they aren't completely useless. Their use is marketing. It gets the word out to developers that this product X is really good! So what if it doesn't translate to real world scenarios or even if the numbers are completely fabricated and you can't even reproduce this under lab conditions. [1] Very few people care enough to look at things that closely. Just seeing a bunch of posts claiming product X is really good is enough to leave a strong impression that product X really is that great. Perception is reality, and perception is usually better influenced by massive claims (even if untrue) rather than realstic iterative progress. Our product is 5% faster than state of the art! just doesn't have that viral headline nature that you need to win over the hearts of the masses. As for why would someone do this. Maybe they don't know better, or maybe they are doing it because they have decided to invest in some technology tribe and thus profit from that tribe surviving, and even more from growing. This is a pretty automatic behavior for humans. Take any tribal war, e.g. XBox One vs PS4. Those who happen to own a XBox One (perhaps as a gift) can be seen at various places passionately arguing that XBox One is better than PS4, even if objectively it has worse hardware and less highly acclaimed exlcusive games. The person is on the XBox One tribe, and working towards getting more users to own XBox One will mean that more developer investments are also made towards the XBox One thanks to the bigger userbase. Thus even if the original claims to get users into the tribe were false, if growth is big enough it may work out well enough at the end. -- [1] The RethinkDB postmortem [2] had a great paragraph about these microbenchmarks. People wanted RethinkDB to be fast on workloads they actually tried, rather than “real world” workloads we suggested. For example, they’d write quick scripts to measure how long it takes to insert ten thousand documents without ever reading them back. MongoDB mastered these workloads brilliantly, while we fought the losing battle of educating the market. [2] http://www.defstartup.org/2017/01/18/why-rethinkdb-failed.html http://www.defstartup.org/2017/01/18/why-rethinkdb-failed.ht...

deleted 10y ago
[deleted]
mcherm 10y ago
Your example from RethinkDB really struck home to me. The idea that superior technology might lose out due to poor marketing or (in this case) a system that is optimized for the real world rather than being optimized for benchmarks really disturbs me. And (this is just my personality) I don't like being disturbed about something without trying to "solve" it. So here's my best thought on how to handle the situation where a team feels that they have a superior product which is losing out to another product that is optimized for benchmarks: > Provide a setting called something like "speed mode". In this mode it is completely optimized for the benchmarks, at the cost of everything else. Default to running without "speed mode", but for anyone who is running benchmarks ask them if they've tried it in "speed mode". A truly competent evaluator will insist on trying the system with the options that are really used in the real world, but then the competent evaluator won't be using an unreliable benchmark anyway. Anyone running the benchmarks just to see how well it works will be likely to turn on something named "speed mode", or at least to do so if asked to. Forums will eventually fill up with people recommending "for real-world loads, you should disable 'speed mode' as it doesn't actually speed them up". Hmm... sounds cool, but I'm not so sure it would actually work. The danger is that you would instead develop a reputation for "cheating" on benchmarks. This is why I'm not very good at marketing.

coldtea 10y ago

>This kind of benchmark is completely useless. On the contrary, that's the only useful type of benchmark. I don't care for "load simulation" full benchmarks, with loads and usage patterns that will be invariably different than mine, and which tell me nothing much. Microbenchmarks on the other hand, are constrained to very specific situations (like the above query), and as such can be very precise in the numbers they give. I know that if I use a similar machine and Node version, and have the same query, I will get the same performance. And that's exactly what programmers use to identify pain points ("hmm, this kind of response handling is slow") and fix it. Isolated and targeted microbenchmarks. I don't care for a "full" benchmark to tell me that "things will slow down with business logic and DB queries". Well, DUH!

xienze 10y ago
> I don't care for a "full" benchmark to tell me that "things will slow down with business logic and DB queries". Well, DUH! Yes but you don't know exactly how much those things will differ between languages, which is important. For instance, you may say "wow, Node smokes Java in this echo server benchmark, I'm sold!", only to later find out that e.g., DB queries run 3x slower in Node than Java. Suddenly a more real-world benchmark makes sense...
- coldtea 10y ago
  >only to later find out that e.g., DB queries run 3x slower in Node than Java. Suddenly a more real-world benchmark makes sense No, then I just need an additional DB query microbenchmark.

jondubois 10y ago

I think it shows that the author(s) behind the library is/are committed to improving performance. Sure, a lot of companies like to publish benchmarks like this to make their product look better than it is (E.g. only showing the good parts), but in this case I know the author and I can vouch that he is independent and uncompromising. You could argue that the baseline performance of a library doesn't matter as much once you start adding lots of custom logic on top, but it's still highly relevant for lightweight workloads (which are actually quite common E.g. Basic chat systems).

enraged_camel 10y ago
>>I think it shows that the author(s) behind the library is/are committed to improving performance. No, it shows that they care about benchmarks, which are rarely if ever representative of real-life scenarios.