4 ms·

I'm so happy this is (currently) the top comment and people are starting to realize measuring perf with these well tuned micro-benchmarks is a sham.

by inglor 4y ago

jwdunne 4y ago
Yes. I was thinking exactly this when coming into the comments. Real world benchmarks take more time to prepare, we get that, but let’s be honest: a bold claim _needs_ bold evidence.
warinukraine 4y ago
Why is it a sham? It's useful to know that x is faster. As a user of x I don't really care if the reason it's faster is it's C++ under the hood. That's really an implementation detail for me.
- lupire 4y ago
  What happens when you use X because it's fast at one thing in a microbenchmark, but it locks you into an environment that is slow at the rest of your program?
- inglor 4y ago
  Because it's measuring a scenario that's not representative of real usages of X and is usually tweaked for the framework/tool the author is trying to showcase. For example: all the deno/bun benchmarks often use the Node.js frameworks in an unnecessarily slow way and don't measure apples to apples. (like using uWS in bun but not uWS for Node in the benchmark). This one in particular just measures the speed of "hello world" in plaintext and JSON with very specific parameters.
- anonymoushn 4y ago
  The real thing will be some mix of your code and the library's code, and in cases like this where you'd be writing all your logic in Python vs Golang, that will have a significant effect on the results. A while back I implemented a game rules engine and MCTS in Torch (the Lua library). The training loop spent like half of its time running the rules engine. Writing it in Python would have been a disaster in comparison. To really make good use of the hardware and spend more time in the optimized machine learning code provided by libraries, one would have to write their rules engine in some language other than Lua or Python.
- fsdjkflsjfsoij 4y ago
  Because the moment you actually write any server or business logic in Python performance drops off a cliff and it doesn't differ significantly from any of the other Python web frameworks.

I'm so happy this is (currently) the top comment and people are starting to realize measuring perf with these well tuned micro-benchmarks is a sham.

by inglor 4y ago

I'm so happy this is (currently) the top comment and people are starting to realize measuring perf with these well tuned micro-benchmarks is a sham.

jwdunne 4y ago

Yes. I was thinking exactly this when coming into the comments. Real world benchmarks take more time to prepare, we get that, but let’s be honest: a bold claim _needs_ bold evidence.

warinukraine 4y ago

Why is it a sham? It's useful to know that x is faster. As a user of x I don't really care if the reason it's faster is it's C++ under the hood. That's really an implementation detail for me.

lupire 4y ago
What happens when you use X because it's fast at one thing in a microbenchmark, but it locks you into an environment that is slow at the rest of your program?
inglor 4y ago
Because it's measuring a scenario that's not representative of real usages of X and is usually tweaked for the framework/tool the author is trying to showcase. For example: all the deno/bun benchmarks often use the Node.js frameworks in an unnecessarily slow way and don't measure apples to apples. (like using uWS in bun but not uWS for Node in the benchmark). This one in particular just measures the speed of "hello world" in plaintext and JSON with very specific parameters.
anonymoushn 4y ago
The real thing will be some mix of your code and the library's code, and in cases like this where you'd be writing all your logic in Python vs Golang, that will have a significant effect on the results. A while back I implemented a game rules engine and MCTS in Torch (the Lua library). The training loop spent like half of its time running the rules engine. Writing it in Python would have been a disaster in comparison. To really make good use of the hardware and spend more time in the optimized machine learning code provided by libraries, one would have to write their rules engine in some language other than Lua or Python.
fsdjkflsjfsoij 4y ago
Because the moment you actually write any server or business logic in Python performance drops off a cliff and it doesn't differ significantly from any of the other Python web frameworks.