3 ms·

Not totally useless. This shows the performance and overhead of the library/framework not the task. Many readers should have a feel for their own use cases and

by rubyfan 10y ago

Not totally useless. This shows the performance and overhead of the library/framework not the task. Many readers should have a feel for their own use cases and be able to relate them to "hello world" benchmark responses, for example I usually take it to mean divide stated performance by 10 immediately if a simple DB query is involved, etc. Also if today you are using one of the compared setups you should know what performance you currently have and what tuning went into it to get a relativity.

Yokohiii 10y ago
This is why they should benchmark against a raw c implementation with same feature set. People seem to try to raise the bar with just ignoring what is ahead of them.
- rubyfan 10y ago
  If you hit the README for the project you'll see they've done comparisons against a number of their peers.
Strom 10y ago
Microbenchmark results don't linearly scale to everything else. Just because language X can print "Hello, World!" 2x faster than language Y doesn't mean that every other operation is also 2x faster. For example a huge factor is algorithm quality. Language X may be fast at "Hello, World!", but then proceed to have QuickSort as its standard sorting function, while language Y has Timsort. [1] Language X may have a nicely optimized C library for hashing, while language Y has AVX2 optimized ASM. Some languages don't even have a wide & well-optimized standard library. Thus you can only really tell how good a language/library is for your usecase if you test with an actual real world scenario. Additionally, the ultimate microbenchmark winning code is one that does every trick in the book while not caring about anything else. This means hooking the kernel, unloading every kernel module / driver that isn't necessary for the microbenchmark, and doing the microbenchmark work at ring0 with absolute minimum overhead. Written in ASM, which is implanted by C code, which is launched by node.js. Then, if there's any data dependant processing in the microbenchmark, the winning code will precompute everything and load the full 2 TB of precomputed data into RAM. The playing field is even, JVM & Apache, or whatever else is the competition will also be run on this 2 TB RAM machine of course. They just won't use it, because they aren't designed to deliver the best results in this single microbenchmark. The point is that, not only don't microbenchmark results mean linear scaling for other work, but the techniques to achieve the microbenchmark results may even be detrimental to everything else! -- [1] For some data sets QuickSort is actually faster. Goes to show you that the best choice is highly dependant on actual use.
- rubyfan 10y ago
  > Microbenchmark results don't linearly scale to everything else. Certainly they don't. But when evaluating something like this it is up to the reader to have critical thinking skills and realistic expectations about the level of experimental design applied to an admittedly alpha implementation published on a wiki on GitHub vs. maybe reading something like published in a peer review journal.
StreamBright 10y ago
Well, there is another aspect to it. Having 2.000.000 clients connected at the same time is much more important scaling factor than we need 1M req/s. Usually people scale services on multiple dimensions (also financial aspects). On the topic of we are just benchmarking the framework, sure, but I would like to also add the test when we tested for other requirements as well. Visualising the results with p50..99.99 latency also would be meaningful.

by rubyfan 10y ago

Yokohiii 10y ago

This is why they should benchmark against a raw c implementation with same feature set. People seem to try to raise the bar with just ignoring what is ahead of them.

rubyfan 10y ago
If you hit the README for the project you'll see they've done comparisons against a number of their peers.

Strom 10y ago

Microbenchmark results don't linearly scale to everything else. Just because language X can print "Hello, World!" 2x faster than language Y doesn't mean that every other operation is also 2x faster. For example a huge factor is algorithm quality. Language X may be fast at "Hello, World!", but then proceed to have QuickSort as its standard sorting function, while language Y has Timsort. [1] Language X may have a nicely optimized C library for hashing, while language Y has AVX2 optimized ASM. Some languages don't even have a wide & well-optimized standard library. Thus you can only really tell how good a language/library is for your usecase if you test with an actual real world scenario. Additionally, the ultimate microbenchmark winning code is one that does every trick in the book while not caring about anything else. This means hooking the kernel, unloading every kernel module / driver that isn't necessary for the microbenchmark, and doing the microbenchmark work at ring0 with absolute minimum overhead. Written in ASM, which is implanted by C code, which is launched by node.js. Then, if there's any data dependant processing in the microbenchmark, the winning code will precompute everything and load the full 2 TB of precomputed data into RAM. The playing field is even, JVM & Apache, or whatever else is the competition will also be run on this 2 TB RAM machine of course. They just won't use it, because they aren't designed to deliver the best results in this single microbenchmark. The point is that, not only don't microbenchmark results mean linear scaling for other work, but the techniques to achieve the microbenchmark results may even be detrimental to everything else! -- [1] For some data sets QuickSort is actually faster. Goes to show you that the best choice is highly dependant on actual use.

rubyfan 10y ago
> Microbenchmark results don't linearly scale to everything else. Certainly they don't. But when evaluating something like this it is up to the reader to have critical thinking skills and realistic expectations about the level of experimental design applied to an admittedly alpha implementation published on a wiki on GitHub vs. maybe reading something like published in a peer review journal.

StreamBright 10y ago

Well, there is another aspect to it. Having 2.000.000 clients connected at the same time is much more important scaling factor than we need 1M req/s. Usually people scale services on multiple dimensions (also financial aspects). On the topic of we are just benchmarking the framework, sure, but I would like to also add the test when we tested for other requirements as well. Visualising the results with p50..99.99 latency also would be meaningful.