8 ms·

Framework Benchmarks Round 19

90 points by bhauer 6y ago

minimaxir 6y ago
Interesting that drogon is relatively new (first release a year ago; new to Round 19) and is utterly destroying the benchmarks. https://github.com/an-tao/drogon https://github.com/an-tao/drogon Show HN: https://news.ycombinator.com/item?id=19427332 https://news.ycombinator.com/item?id=19427332
- umvi 6y ago
  Well, it's hard for high level scripting languages to compete with C++ performance-wise... I would have liked to see some other C++ ones on there, like Crow[0] [0] https://github.com/ipkn/crow https://github.com/ipkn/crow
  - continuations 6y ago
    Last committed 2017. Looks like it's dead?
- marricks 6y ago
  I heard some old guard person at Mozilla talk about these benchmarks last year when Rust's ACTIX was on top, something like "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest", at the time I took it to mean ACTIX being the fastest has more to do with it being new then it being written in rust. Given a newcomer is at the top again that seems to ring true? Does seem odd older frameworks wouldn't keep up with that game, but perhaps once they get the publicity and user base they can worry more about features than having the highest benchmark?
  - ori_b 6y ago
    People ask for more features over time. I'd expect older frameworks to be more featureful, but slower.
  - Klonoar 6y ago
    It's worth noting that actix-web is at the top by doing some insane tricks, which you wouldn't normally do in writing a web application. e.g at one point they used Askama as their templating engine (may have switched), which isn't really fair as using Askama means you force a recompile of your project for each template change - and I can tell you from experience that your compile times will shoot through the roof as well. Nobody is going to bother doing that. I say all of this as an actix-web user. It's my go-to. I just view these benchmarks as "if I ever needed to, I know I can wrangle performance out of it... but I won't assume it by default".
    - terhechte 6y ago
      But most other projects on there do that as well. I regard tech empower as the maximum performance I can get with a given framework if need be. With actin I can add Askama, with Vapor, there's no way to make it faster. As a side note, Ramshorn is even faster than Askama and does not need compiled templates: https://github.com/maciejhirsz/ramhorns https://github.com/maciejhirsz/ramhorns
      - Klonoar 6y ago
        Regarding your first sentence... I think we have the same conclusion. :) Thanks for the heads up re: Ramhorns. I'd seen it before but Mustache templates aren't particularly my thing - Django/Python guy in a previous life, so I've got love for the templating models from there. Willing to sacrifice a small amount of speed for developer ergonomics, things almost universally get rewritten once big enough scale comes into play anyway.
    - chrismorgan 6y ago
      I don’t think recompiling for template changes is as unreasonable as you depict. It’s a standard technique in various languages, though I acknowledge that they’re mostly not languages that compile to native code, but rather an IL (e.g. ASP.NET with Razor) where the compilation time is somewhat lower. For a language like Rust, compilation times and tooling support mean that the technique is perhaps not ready for prime time, but it’s fundamentally a perfectly sound technique, and can produce wondrous developer ergonomics (again I cite Razor). There’s something quite incongruous about using dynamically-typed and basically uncheckable template language when you’re doing the rest of the code in Rust.
  - chiefsucker 6y ago
    > "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest" I’m not sure I understand this, i. e. TRANTOR (which is the underlying network library for Drogon) recently merged in a patch that optimizes memory handling in a low level component of its event loop where object construction was minimized (first tests show another performance increase of about 5–10% on top of the current benchmark results). I fail to understand how such improvements have something to do with “all the latest methods and libraries”?
- Isomorpheus 6y ago
  What's impressing to me is that in the most "IO-bottlenecked" benchmarks: 'Multiple queries' and 'Data updates' (on Cloud) Drogon is smoking the next best frameworks. What's up with that?
  - orthoxerox 6y ago
    Postgres libs for C++ are probably very good at async.
- chiefsucker 6y ago
  One of Drogon’s project goals is great performance. That’s also one of the reasons why it’s written in C++.
bsaul 6y ago
I was surprised to see the performance of php frameworks. Has there been something special happening with this language ? It looks completely on top of all the other dynamic languages on most benchmarks, which i really don’t understand.
- open-source-ux 6y ago
  PHP 7.0 (which was released in Dec 2015) featured a new core with dramatic improvement in performance over PHP 5.X. PHP 8 is due for release in Dec 2020 and will feature a JIT compiler which is expected to improve performance even further. Regardless of what people think of PHP, it's nice to see the PHP community tackle performace and resource usage as a major priority. A contrast to parts of the development community who seem to care very little for performance and resource usage.
- jorams 6y ago
  It's important to note that all the PHP results that are high up on the lists are using Workerman[1] with the event module, Swoole[2] or ngx_php[3]. They don't use the traditional PHP runtime model, which boots up and tears down the world for every request. Global variables, which are no problem in "normal" PHP code, are going to cause serious trouble in these environments. They are essentially gluing a very fast async IO framework to a fairly minimal amount of PHP application code. The vast, vast majority of PHP code you will find in the wild is running a more traditional stack, and those results are much lower down the list. The results are still impressive though. PHP is very good at gluing fast C code together without slowing it down much. Especially Workerman is impressive, since it appears to still run a significant amount of PHP code. [1]: https://github.com/walkor/Workerman https://github.com/walkor/Workerman [2]: https://www.swoole.co.uk/ https://www.swoole.co.uk/ [3]: https://github.com/rryqszq4/ngx_php7 https://github.com/rryqszq4/ngx_php7
  - gremlinsinc 6y ago
    Part of that's also that php 7.4 is much faster, so of course that alone would improve it more, I wonder what JIT will do when 8.0 launches in December. I use swoole w/ laravel, still trying to figure out best ways to milk it for more performance, and also grok the way it handles async and passes variables around using swoole_Table.
jph 6y ago
Rocket (Rust) is the framework on my new project, and the benchmarks clearly show that speed improvements would be very welcome. If anyone here wants to work on this area, please contact me and I'll be glad to boost the Rocket numbers together.
- _bxg1 6y ago
  How does Rocket compare with actix on other dimensions? (i.e., what made you decide to pick it for your project?)
  - jph 6y ago
    Broadly, Rocket and Actix are both great in my opinion. I use actix-web on another project. IMHO Actix is conceptually sophisticated in a good way, has great async capabilties, great speed, and runs on Rust stable which is necessary for that project. However, actix-web had a continuity problem when the leader stepped away - see HN if you want to know more about this. So we decided to try Rocket for this new project. This project involves more junior developers, has no special speed needs, and is fine on Rust nightly for now. The Rocket guides are excellent, especially for onboarding. The concepts feel simpler, which is fine for this project. We're eager for Rocket to launch its async version and its Rust stable version. Ultimately the bulk of the code is similar, because both projects are web apps that use Diesel, Serde, reqwest, and other typical crates. We're looking to the future to extract some of the code into framework-independent crates, such as our API structs, business logic, and the like.
    - _bxg1 6y ago
      Thanks! I'm very curious about the Rust web space even though it seems to still be in a formative stage. It sounds like you guys are really on the cutting edge
    - freeopinion 6y ago
      I'm just gonna say here that Rocket is never going to run in stable. I write this specifically because I'm never right and I hope that my writing this publicly will lead to Rocket's immediate release on Rust stable just to prove me wrong. And here's a cheer for those doing the work to make it async.
tfparty2 6y ago
The addition of the "realistic" vs "synthetic" benchmarks is very welcome, with "realistic" being the default. Hopefully this could help get things to a better state. Still questionable as to what is allowed in "realistic" though. Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic". Perhaps there needs to be a "marketed" category, for use as the docs describe. For aspnet core, this would include using entity framework. Not much point in splitting the categories if it isn't a uniformly split thing. It'd be nice to see benchmarks where: • Everything was implemented as documented in the framework's tutorials. • Realistic things were included end-to-end (payload validation, csrf (if applicable), etc). Still seems like a whole basket of fruit comparisons here.
- dangerboysteve 6y ago
  The hardware physical/cloud toggle is also interesting.
- nodamage 6y ago
  > Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic". Why not? I don't think using raw SQL instead of an ORM is particularly uncommon. The "platform" benchmarks (aspcore-rhtx-pg, aspcore-ado-pg) that appear to skip the ASP.NET Core framework entirely and write raw HTTP responses are definitely not realistic though. I'm going to assume that these are mislabeled because this filter is new and not all the benchmarks have been updated yet.
  - tfparty2 6y ago
    I'll agree that raw database queries aren't unheard of, but EF and Dapper are much more suggested (and IMO "realistic") database paths for "asp.net core" additionally, I'm not quite sure that this is the recommended way to return database results: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/Benchmarks/Middleware/SingleQueryRawMiddleware.cs#L33 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... What I believe is more "realistic" is to: • Use the framework as described: https://docs.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-3.1&tabs=visual-studio#return-values https://docs.microsoft.com/en-us/aspnet/core/tutorials/first... To be fair, I'm picking on aspnet core because I like it so much (via F#). The framework is great, but this gaming of public benchmarks (techempower, benchmarksgame) by "breaking the same rules as everyone else" rather than investing time into making these results more useful is disappointing.
dom96 6y ago
If you're thinking of evaluating language performance by looking at these benchmarks then think again. A lot of the benchmarks here (h2o.cr and pico.v being prime examples) are thin wrappers over libraries written in C/C++.
- giulianob 6y ago
  That's what's actually fairly impressive about C# being #1 on plaintext. Both ASP.NET and Kestrel (the frontend server) are both fully in managed code.
  - jonathanoliver 6y ago
    Various teams at Microsoft should be congratulated for their efforts. A few years back, ASP.NET was last (or near last) on this benchmark for simple, plaintext requests. They've come a long way and put some serious engineering into their system.
    - sebastienros 6y ago
      You can't even imagine. A goal has been set by the management to get specific results on some benchmarks, by improving the framework. We have weekly meetings to follow up on improvements and next tasks to keep track on these goals. And these meetings involve super qualified engineers and also brilliant open source contributors. I can t describe the level of technicity at each of these meetings. The impact on the framework is tremendous, you can follow it as our charts are public. And the goals are already reached, which is not yet reflected on TechEmpower as we only submit release bits. This is good for PR, but as much for also for any developper using dotnet.
      - jonathanoliver 6y ago
        I wrote this article 6 years ago and it got quite a bit of traction here on HN and Reddit: https://blog.jonathanoliver.com/why-i-left-dot-net/ https://blog.jonathanoliver.com/why-i-left-dot-net/ Virtually every single item on my list is now completely irrelevant. Although I'm not a daily developer using .NET anymore, I am astounded that such a turnaround was even possible. My hat is off to you and the teams you work with. Also, for the record C# is still a phenomenal language in my book.
kumarvvr 6y ago
I wish there was an additional set of data that uses this set, and then computes the average cost of using each framework on various cloud providers. Its amazing to see python based stuff so low in the benchmarks, yet so high in general popularity. In cloud based servers, where requests per second directly translates to inverse USD, its imperative costs are incorporated into the selection process for the software stack.
- ryuukk_ 6y ago
  this
- p_l 6y ago
  There are some writings by people from Stackoverflow on how much they saved going with native COMPILING NET stack.
ssijak 6y ago
People like bashing java, but java based frameworks (and other JVM based ones) are always at the top. Solid foundations, great virtual machine, epic tooling, great and reliable libraries (which do not change every 3 months like in node), and in the recent time much faster evolution of the language. And if you do not like java as a language (which has its downfalls and problems) you can use Kotlin or Scala, or even Haskell on the JVM if you fancy so. Funny to see rails, django, laravel and other most popular frameworks from dynamic languages at the bottom of the list with such bad performance. Even the Java heavy weight Spring framework is much closer to the top. Not to mention that JVM can support all kinds of concurrent, parallelism and scaling models.
- hombre_fatal 6y ago
  I'd say performance in this benchmark is inversely related with how nice it is to express business logic in the respective language/framework, and that's a pretty big deal. For example, https://github.com/an-tao/drogon https://github.com/an-tao/drogon (benchmark winner) or https://oci-pronghorn.gitbook.io/greenlightning/ https://oci-pronghorn.gitbook.io/greenlightning/ just aren't pleasant to work with. For example, compare Green Lightning with http://sparkjava.com/ http://sparkjava.com/ and look at how much slower Spark is while clearly being nicer to use. I worked with Netty at work for two years and it was a dream to use something like Express or Flask on the side, not having to worry sharing/disposing ByteBufs, not dealing with the weird in-channel/out-channel abstraction (vs simple middleware), not investigating how async-/thread-safe a given library was, etc. It just all goes back to trade-offs. You're going to have different considerations if you're building a Caddy competitor vs a complex game server. Another consideration is how horizontally scalable you can be which is going to be different for a forum vs a websocket server.
  - randombytes6869 6y ago
    Vert.X is a relatively light layer on top of Netty and it is quite nice to work with. Spring WebFlux is also built on top of Netty, and missing some benchmarks, but I have no reason to think it would perform poorly. Its very easy to use for anyone with Spring experience. The fastest frameworks are all nonblocking, like Express. They use streaming and reactive patterns, if it wasn't for Java's types everywhere some people wouldn't be able to tell the difference between Vert.X using RxJava and Express. The main design difference is that JS does non-blocking on a single core vs Java which does thread-per-core. Basically, in Java you can scale out non-blocking across all cores and still share memory when you want to. If you don't want to deal with thread safety and non-blocking DropWizard performs quite well using traditional threads. Just saying, performance and easy of use are not always exclusive. Maybe Vert.X didn't exist back when you were using Netty but these days it would be unthinkable to use Netty itself, the abstraction Vert.X provides is far easier to use and you don't really lose any performance. Java's bad performance reputation is 90% due to old versions of Spring. Spring Boot has come a huge way, with performance maybe 100X better than Spring 4. Spring WebFlux takes this even farther with a Vert.X-like api, probably reaching near-Netty performance
gremlinsinc 6y ago
Damn, I just settled on a high performance framework for my next reddit killer meets slack clone + SaaS, and now I need to rethink that and add more months to actually launching, on the off-chance my app takes off and needs to quickly support millions of connections at once.
- deathtrader666 6y ago
  Stop reading my mind /s
  - gremlinsinc 6y ago
    haha, I do this every few months on a side project, then usually fall back to laravel+vue just cause it's what I know. Though, at least now swoole gives me a little more 'oomph' w/ php.
viig99 6y ago
Modern c++ is surprisingly easy to pickup especially for developers coming from other languages, very easy to focus on the good parts and totally disregard the messy bits, working with older libraries is still a pain though.
Spiritus 6y ago
What does the new "<T>" icon represent on the graphs (only present on some of the frameworks)?
- Spiritus 6y ago
  Answering my own question: > TPR-3 is a composite hardware environment score for a three-machine configuration, derived from all test types for TPR-tagged <T> frameworks. (More about TPR frameworks and TPR-1 versus TPR-3)[1] [1] https://github.com/TechEmpower/FrameworkBenchmarks/wiki/TechEmpower-Performance-Rating-(TPR)#two-flavors-tpr-1-and-tpr-3 https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Tech...
  - bhauer 6y ago
    Indeed, this is related to our (experimental) addition of a performance rating for the hardware environment we're testing on. This is rendered as the big number on the composite scores page. Theoretically, our benchmarks covering a diverse spectrum of technologies could be useful for measuring hardware and network performance. But a full run of all test implementations takes 111 hours, and I suspect few people who want to measure hardware performance would be willing to wait that long. So we selected a subset to use for hardware measurement, hence the icon you see.
matt42 6y ago
I wrote a new C++ framework (lithium) last december ranked #4 in this benchmark and while this benchmark does a great job comparing raw performances of many frameworks on a set of test, here are my thoughts: 1/ Best Frameworks are all using Non blocking network IO (based on linux epoll or equivalents) 2/ Very few companies need such performances. Don't look for those number if you have less than 10M visitor/months 3/ Tuning the number of sql connections can do x5 on performances and all framework did not tune this parameter. 4/ Even if you a low ranked framework, it may not be the bottleneck of your stack so changing to a top performer may not change your thoughput overall.
Zingler 6y ago
A fun diversion, but largely meaningless.
barbarbar 6y ago
Just came to say that it is an amazing and extremely thorough job they have done in all these runs. And also still mind-blowing how much work the fastest frameworks can do.

90 points by bhauer 6y ago

Interesting that drogon is relatively new (first release a year ago; new to Round 19) and is utterly destroying the benchmarks. https://github.com/an-tao/drogon https://github.com/an-tao/drogon Show HN: https://news.ycombinator.com/item?id=19427332 https://news.ycombinator.com/item?id=19427332

umvi 6y ago
Well, it's hard for high level scripting languages to compete with C++ performance-wise... I would have liked to see some other C++ ones on there, like Crow[0] [0] https://github.com/ipkn/crow https://github.com/ipkn/crow
- continuations 6y ago
  Last committed 2017. Looks like it's dead?
marricks 6y ago
I heard some old guard person at Mozilla talk about these benchmarks last year when Rust's ACTIX was on top, something like "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest", at the time I took it to mean ACTIX being the fastest has more to do with it being new then it being written in rust. Given a newcomer is at the top again that seems to ring true? Does seem odd older frameworks wouldn't keep up with that game, but perhaps once they get the publicity and user base they can worry more about features than having the highest benchmark?
- ori_b 6y ago
  People ask for more features over time. I'd expect older frameworks to be more featureful, but slower.
- Klonoar 6y ago
  It's worth noting that actix-web is at the top by doing some insane tricks, which you wouldn't normally do in writing a web application. e.g at one point they used Askama as their templating engine (may have switched), which isn't really fair as using Askama means you force a recompile of your project for each template change - and I can tell you from experience that your compile times will shoot through the roof as well. Nobody is going to bother doing that. I say all of this as an actix-web user. It's my go-to. I just view these benchmarks as "if I ever needed to, I know I can wrangle performance out of it... but I won't assume it by default".
  - terhechte 6y ago
    But most other projects on there do that as well. I regard tech empower as the maximum performance I can get with a given framework if need be. With actin I can add Askama, with Vapor, there's no way to make it faster. As a side note, Ramshorn is even faster than Askama and does not need compiled templates: https://github.com/maciejhirsz/ramhorns https://github.com/maciejhirsz/ramhorns
    - Klonoar 6y ago
      Regarding your first sentence... I think we have the same conclusion. :) Thanks for the heads up re: Ramhorns. I'd seen it before but Mustache templates aren't particularly my thing - Django/Python guy in a previous life, so I've got love for the templating models from there. Willing to sacrifice a small amount of speed for developer ergonomics, things almost universally get rewritten once big enough scale comes into play anyway.
  - chrismorgan 6y ago
    I don’t think recompiling for template changes is as unreasonable as you depict. It’s a standard technique in various languages, though I acknowledge that they’re mostly not languages that compile to native code, but rather an IL (e.g. ASP.NET with Razor) where the compilation time is somewhat lower. For a language like Rust, compilation times and tooling support mean that the technique is perhaps not ready for prime time, but it’s fundamentally a perfectly sound technique, and can produce wondrous developer ergonomics (again I cite Razor). There’s something quite incongruous about using dynamically-typed and basically uncheckable template language when you’re doing the rest of the code in Rust.
- chiefsucker 6y ago
  > "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest" I’m not sure I understand this, i. e. TRANTOR (which is the underlying network library for Drogon) recently merged in a patch that optimizes memory handling in a low level component of its event loop where object construction was minimized (first tests show another performance increase of about 5–10% on top of the current benchmark results). I fail to understand how such improvements have something to do with “all the latest methods and libraries”?
Isomorpheus 6y ago
What's impressing to me is that in the most "IO-bottlenecked" benchmarks: 'Multiple queries' and 'Data updates' (on Cloud) Drogon is smoking the next best frameworks. What's up with that?
- orthoxerox 6y ago
  Postgres libs for C++ are probably very good at async.
chiefsucker 6y ago
One of Drogon’s project goals is great performance. That’s also one of the reasons why it’s written in C++.

bsaul 6y ago

I was surprised to see the performance of php frameworks. Has there been something special happening with this language ? It looks completely on top of all the other dynamic languages on most benchmarks, which i really don’t understand.

open-source-ux 6y ago
PHP 7.0 (which was released in Dec 2015) featured a new core with dramatic improvement in performance over PHP 5.X. PHP 8 is due for release in Dec 2020 and will feature a JIT compiler which is expected to improve performance even further. Regardless of what people think of PHP, it's nice to see the PHP community tackle performace and resource usage as a major priority. A contrast to parts of the development community who seem to care very little for performance and resource usage.
jorams 6y ago
It's important to note that all the PHP results that are high up on the lists are using Workerman[1] with the event module, Swoole[2] or ngx_php[3]. They don't use the traditional PHP runtime model, which boots up and tears down the world for every request. Global variables, which are no problem in "normal" PHP code, are going to cause serious trouble in these environments. They are essentially gluing a very fast async IO framework to a fairly minimal amount of PHP application code. The vast, vast majority of PHP code you will find in the wild is running a more traditional stack, and those results are much lower down the list. The results are still impressive though. PHP is very good at gluing fast C code together without slowing it down much. Especially Workerman is impressive, since it appears to still run a significant amount of PHP code. [1]: https://github.com/walkor/Workerman https://github.com/walkor/Workerman [2]: https://www.swoole.co.uk/ https://www.swoole.co.uk/ [3]: https://github.com/rryqszq4/ngx_php7 https://github.com/rryqszq4/ngx_php7
- gremlinsinc 6y ago
  Part of that's also that php 7.4 is much faster, so of course that alone would improve it more, I wonder what JIT will do when 8.0 launches in December. I use swoole w/ laravel, still trying to figure out best ways to milk it for more performance, and also grok the way it handles async and passes variables around using swoole_Table.

jph 6y ago

Rocket (Rust) is the framework on my new project, and the benchmarks clearly show that speed improvements would be very welcome. If anyone here wants to work on this area, please contact me and I'll be glad to boost the Rocket numbers together.

_bxg1 6y ago
How does Rocket compare with actix on other dimensions? (i.e., what made you decide to pick it for your project?)
- jph 6y ago
  Broadly, Rocket and Actix are both great in my opinion. I use actix-web on another project. IMHO Actix is conceptually sophisticated in a good way, has great async capabilties, great speed, and runs on Rust stable which is necessary for that project. However, actix-web had a continuity problem when the leader stepped away - see HN if you want to know more about this. So we decided to try Rocket for this new project. This project involves more junior developers, has no special speed needs, and is fine on Rust nightly for now. The Rocket guides are excellent, especially for onboarding. The concepts feel simpler, which is fine for this project. We're eager for Rocket to launch its async version and its Rust stable version. Ultimately the bulk of the code is similar, because both projects are web apps that use Diesel, Serde, reqwest, and other typical crates. We're looking to the future to extract some of the code into framework-independent crates, such as our API structs, business logic, and the like.
  - _bxg1 6y ago
    Thanks! I'm very curious about the Rust web space even though it seems to still be in a formative stage. It sounds like you guys are really on the cutting edge
  - freeopinion 6y ago
    I'm just gonna say here that Rocket is never going to run in stable. I write this specifically because I'm never right and I hope that my writing this publicly will lead to Rocket's immediate release on Rust stable just to prove me wrong. And here's a cheer for those doing the work to make it async.

tfparty2 6y ago

The addition of the "realistic" vs "synthetic" benchmarks is very welcome, with "realistic" being the default. Hopefully this could help get things to a better state. Still questionable as to what is allowed in "realistic" though. Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic". Perhaps there needs to be a "marketed" category, for use as the docs describe. For aspnet core, this would include using entity framework. Not much point in splitting the categories if it isn't a uniformly split thing. It'd be nice to see benchmarks where: • Everything was implemented as documented in the framework's tutorials. • Realistic things were included end-to-end (payload validation, csrf (if applicable), etc). Still seems like a whole basket of fruit comparisons here.

dangerboysteve 6y ago
The hardware physical/cloud toggle is also interesting.
nodamage 6y ago
> Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic". Why not? I don't think using raw SQL instead of an ORM is particularly uncommon. The "platform" benchmarks (aspcore-rhtx-pg, aspcore-ado-pg) that appear to skip the ASP.NET Core framework entirely and write raw HTTP responses are definitely not realistic though. I'm going to assume that these are mislabeled because this filter is new and not all the benchmarks have been updated yet.
- tfparty2 6y ago
  I'll agree that raw database queries aren't unheard of, but EF and Dapper are much more suggested (and IMO "realistic") database paths for "asp.net core" additionally, I'm not quite sure that this is the recommended way to return database results: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/Benchmarks/Middleware/SingleQueryRawMiddleware.cs#L33 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... What I believe is more "realistic" is to: • Use the framework as described: https://docs.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-3.1&tabs=visual-studio#return-values https://docs.microsoft.com/en-us/aspnet/core/tutorials/first... To be fair, I'm picking on aspnet core because I like it so much (via F#). The framework is great, but this gaming of public benchmarks (techempower, benchmarksgame) by "breaking the same rules as everyone else" rather than investing time into making these results more useful is disappointing.

dom96 6y ago

If you're thinking of evaluating language performance by looking at these benchmarks then think again. A lot of the benchmarks here (h2o.cr and pico.v being prime examples) are thin wrappers over libraries written in C/C++.

giulianob 6y ago
That's what's actually fairly impressive about C# being #1 on plaintext. Both ASP.NET and Kestrel (the frontend server) are both fully in managed code.
- jonathanoliver 6y ago
  Various teams at Microsoft should be congratulated for their efforts. A few years back, ASP.NET was last (or near last) on this benchmark for simple, plaintext requests. They've come a long way and put some serious engineering into their system.
  - sebastienros 6y ago
    You can't even imagine. A goal has been set by the management to get specific results on some benchmarks, by improving the framework. We have weekly meetings to follow up on improvements and next tasks to keep track on these goals. And these meetings involve super qualified engineers and also brilliant open source contributors. I can t describe the level of technicity at each of these meetings. The impact on the framework is tremendous, you can follow it as our charts are public. And the goals are already reached, which is not yet reflected on TechEmpower as we only submit release bits. This is good for PR, but as much for also for any developper using dotnet.
    - jonathanoliver 6y ago
      I wrote this article 6 years ago and it got quite a bit of traction here on HN and Reddit: https://blog.jonathanoliver.com/why-i-left-dot-net/ https://blog.jonathanoliver.com/why-i-left-dot-net/ Virtually every single item on my list is now completely irrelevant. Although I'm not a daily developer using .NET anymore, I am astounded that such a turnaround was even possible. My hat is off to you and the teams you work with. Also, for the record C# is still a phenomenal language in my book.

kumarvvr 6y ago

I wish there was an additional set of data that uses this set, and then computes the average cost of using each framework on various cloud providers. Its amazing to see python based stuff so low in the benchmarks, yet so high in general popularity. In cloud based servers, where requests per second directly translates to inverse USD, its imperative costs are incorporated into the selection process for the software stack.

ryuukk_ 6y ago
this
p_l 6y ago
There are some writings by people from Stackoverflow on how much they saved going with native COMPILING NET stack.

ssijak 6y ago

People like bashing java, but java based frameworks (and other JVM based ones) are always at the top. Solid foundations, great virtual machine, epic tooling, great and reliable libraries (which do not change every 3 months like in node), and in the recent time much faster evolution of the language. And if you do not like java as a language (which has its downfalls and problems) you can use Kotlin or Scala, or even Haskell on the JVM if you fancy so. Funny to see rails, django, laravel and other most popular frameworks from dynamic languages at the bottom of the list with such bad performance. Even the Java heavy weight Spring framework is much closer to the top. Not to mention that JVM can support all kinds of concurrent, parallelism and scaling models.

hombre_fatal 6y ago
I'd say performance in this benchmark is inversely related with how nice it is to express business logic in the respective language/framework, and that's a pretty big deal. For example, https://github.com/an-tao/drogon https://github.com/an-tao/drogon (benchmark winner) or https://oci-pronghorn.gitbook.io/greenlightning/ https://oci-pronghorn.gitbook.io/greenlightning/ just aren't pleasant to work with. For example, compare Green Lightning with http://sparkjava.com/ http://sparkjava.com/ and look at how much slower Spark is while clearly being nicer to use. I worked with Netty at work for two years and it was a dream to use something like Express or Flask on the side, not having to worry sharing/disposing ByteBufs, not dealing with the weird in-channel/out-channel abstraction (vs simple middleware), not investigating how async-/thread-safe a given library was, etc. It just all goes back to trade-offs. You're going to have different considerations if you're building a Caddy competitor vs a complex game server. Another consideration is how horizontally scalable you can be which is going to be different for a forum vs a websocket server.
- randombytes6869 6y ago
  Vert.X is a relatively light layer on top of Netty and it is quite nice to work with. Spring WebFlux is also built on top of Netty, and missing some benchmarks, but I have no reason to think it would perform poorly. Its very easy to use for anyone with Spring experience. The fastest frameworks are all nonblocking, like Express. They use streaming and reactive patterns, if it wasn't for Java's types everywhere some people wouldn't be able to tell the difference between Vert.X using RxJava and Express. The main design difference is that JS does non-blocking on a single core vs Java which does thread-per-core. Basically, in Java you can scale out non-blocking across all cores and still share memory when you want to. If you don't want to deal with thread safety and non-blocking DropWizard performs quite well using traditional threads. Just saying, performance and easy of use are not always exclusive. Maybe Vert.X didn't exist back when you were using Netty but these days it would be unthinkable to use Netty itself, the abstraction Vert.X provides is far easier to use and you don't really lose any performance. Java's bad performance reputation is 90% due to old versions of Spring. Spring Boot has come a huge way, with performance maybe 100X better than Spring 4. Spring WebFlux takes this even farther with a Vert.X-like api, probably reaching near-Netty performance

gremlinsinc 6y ago

Damn, I just settled on a high performance framework for my next reddit killer meets slack clone + SaaS, and now I need to rethink that and add more months to actually launching, on the off-chance my app takes off and needs to quickly support millions of connections at once.

deathtrader666 6y ago
Stop reading my mind /s
- gremlinsinc 6y ago
  haha, I do this every few months on a side project, then usually fall back to laravel+vue just cause it's what I know. Though, at least now swoole gives me a little more 'oomph' w/ php.

viig99 6y ago

Modern c++ is surprisingly easy to pickup especially for developers coming from other languages, very easy to focus on the good parts and totally disregard the messy bits, working with older libraries is still a pain though.

Spiritus 6y ago

What does the new "<T>" icon represent on the graphs (only present on some of the frameworks)?

Spiritus 6y ago
Answering my own question: > TPR-3 is a composite hardware environment score for a three-machine configuration, derived from all test types for TPR-tagged <T> frameworks. (More about TPR frameworks and TPR-1 versus TPR-3)[1] [1] https://github.com/TechEmpower/FrameworkBenchmarks/wiki/TechEmpower-Performance-Rating-(TPR)#two-flavors-tpr-1-and-tpr-3 https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Tech...
- bhauer 6y ago
  Indeed, this is related to our (experimental) addition of a performance rating for the hardware environment we're testing on. This is rendered as the big number on the composite scores page. Theoretically, our benchmarks covering a diverse spectrum of technologies could be useful for measuring hardware and network performance. But a full run of all test implementations takes 111 hours, and I suspect few people who want to measure hardware performance would be willing to wait that long. So we selected a subset to use for hardware measurement, hence the icon you see.

matt42 6y ago

I wrote a new C++ framework (lithium) last december ranked #4 in this benchmark and while this benchmark does a great job comparing raw performances of many frameworks on a set of test, here are my thoughts: 1/ Best Frameworks are all using Non blocking network IO (based on linux epoll or equivalents) 2/ Very few companies need such performances. Don't look for those number if you have less than 10M visitor/months 3/ Tuning the number of sql connections can do x5 on performances and all framework did not tune this parameter. 4/ Even if you a low ranked framework, it may not be the bottleneck of your stack so changing to a top performer may not change your thoughput overall.

Zingler 6y ago

A fun diversion, but largely meaningless.

barbarbar 6y ago

Just came to say that it is an amazing and extremely thorough job they have done in all these runs. And also still mind-blowing how much work the fastest frameworks can do.