4 ms·

tfbvis: Visualizing TechEmpower benchmarks beyond requests per second

2 points by emmanueloga_ 6y ago

emmanueloga_ 6y ago
TechEmpower wants to "provide representative performance measures across a wide field of web application frameworks." Do you think it succeeded in this regard? It seems like the results are pretty well received by the community in general, or at least I haven't seen a lot of people disregard these benchmarks or give them the CLBG treatment [1]. 1: https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- igouy 6y ago
  > at least I haven't seen a lot of people disregard these benchmarks https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_phoenix_compare_to_fastest_frameworks/g3mdkru/ https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_ph... > the CLBG treatment Please say what you mean ?
  - emmanueloga_ 6y ago
    The "The Computer Language Benchmarks Game", if I remember it right, did not start with that title, and the very authors have progressively made more and more emphasis in the fact that the programs are "toy benchmarks" and not useful to compare which language is "fastest". TechEmpower benchmarks, on the other hand, wants to "provide representative performance measures across a wide field of web application frameworks" (and proglang is a prominent part of the framework) so it seems away from the CLBG stance.
    - igouy 6y ago
      Fair enough. > which language is "fastest" Isn't that question too loose to be useful? Isn't it necessary to show that performance measures are representative if we want to claim they are?
    - igouy 6y ago
      > more and more emphasis Well, more and more literal / less and less sarcastic. Consider — "Some people have suggested that I summarize the results, or declare a winner. In order to satisfy this request, I have come up with a unique and subtle quantification system to score languages on their overall performance, which I call the Completely Random and Arbitrary Point System!, or CRAPS![TM], for short. … you can link this page (with properly selected CGI parameters) from your language advocacy page as the final proof of your language's supremacy! Think of the glory." https://web.archive.org/web/20010302143046/http://www.bagley.org/~doug/shootout/craps.shtml https://web.archive.org/web/20010302143046/http://www.bagley...
emmanueloga_ 6y ago
Found tfbvis because I wanted to see more than RPS: specially memory footprint. The headers sortable and can be filtered. The most memory efficient results belong to C, C++, Rust and Go, usually using around 1,1.5GB of RAM max. Java programs in the same ballpark RPS use maybe up to 4 times that much. Not sure how much weight to put in these benchmarks though. After all, they measure performance under stress... How many web apps receive > 200,000 RPS during sustained time? Is there a clever way to extrapolate an estimation of RAM usage under normal load?. Would be cool to see how the memory usage grows as the RPS grow, for different platforms, but I don't think the raw data has that info.
- emmanueloga_ 6y ago
  Hmm was just talking with the author of jooby and its benchmark, he pointed out he just copied one of the docker configs [1] from someone else, which explains why a lot of the benchmarks have similar memory profiles. The jooby author also said he deployed this particular framework with as little as 512MB, which sounds reasonable. So a bunch of JVM implementations may be using "java -Xms4g -Xmx4g", so perhaps the results are not so useful, at least when it comes to the JVM. 1: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/jooby/jooby.dockerfile#L12 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

2 points by emmanueloga_ 6y ago

TechEmpower wants to "provide representative performance measures across a wide field of web application frameworks." Do you think it succeeded in this regard? It seems like the results are pretty well received by the community in general, or at least I haven't seen a lot of people disregard these benchmarks or give them the CLBG treatment [1]. 1: https://benchmarksgame-team.pages.debian.net/benchmarksgame/why-measure-toy-benchmark-programs.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

igouy 6y ago
> at least I haven't seen a lot of people disregard these benchmarks https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_phoenix_compare_to_fastest_frameworks/g3mdkru/ https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_ph... > the CLBG treatment Please say what you mean ?
- emmanueloga_ 6y ago
  The "The Computer Language Benchmarks Game", if I remember it right, did not start with that title, and the very authors have progressively made more and more emphasis in the fact that the programs are "toy benchmarks" and not useful to compare which language is "fastest". TechEmpower benchmarks, on the other hand, wants to "provide representative performance measures across a wide field of web application frameworks" (and proglang is a prominent part of the framework) so it seems away from the CLBG stance.
  - igouy 6y ago
    Fair enough. > which language is "fastest" Isn't that question too loose to be useful? Isn't it necessary to show that performance measures are representative if we want to claim they are?
  - igouy 6y ago
    > more and more emphasis Well, more and more literal / less and less sarcastic. Consider — "Some people have suggested that I summarize the results, or declare a winner. In order to satisfy this request, I have come up with a unique and subtle quantification system to score languages on their overall performance, which I call the Completely Random and Arbitrary Point System!, or CRAPS![TM], for short. … you can link this page (with properly selected CGI parameters) from your language advocacy page as the final proof of your language's supremacy! Think of the glory." https://web.archive.org/web/20010302143046/http://www.bagley.org/~doug/shootout/craps.shtml https://web.archive.org/web/20010302143046/http://www.bagley...

emmanueloga_ 6y ago

Found tfbvis because I wanted to see more than RPS: specially memory footprint. The headers sortable and can be filtered. The most memory efficient results belong to C, C++, Rust and Go, usually using around 1,1.5GB of RAM max. Java programs in the same ballpark RPS use maybe up to 4 times that much. Not sure how much weight to put in these benchmarks though. After all, they measure performance under stress... How many web apps receive > 200,000 RPS during sustained time? Is there a clever way to extrapolate an estimation of RAM usage under normal load?. Would be cool to see how the memory usage grows as the RPS grow, for different platforms, but I don't think the raw data has that info.

emmanueloga_ 6y ago
Hmm was just talking with the author of jooby and its benchmark, he pointed out he just copied one of the docker configs [1] from someone else, which explains why a lot of the benchmarks have similar memory profiles. The jooby author also said he deployed this particular framework with as little as 512MB, which sounds reasonable. So a bunch of JVM implementations may be using "java -Xms4g -Xmx4g", so perhaps the results are not so useful, at least when it comes to the JVM. 1: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/jooby/jooby.dockerfile#L12 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...