12 ms·

TechEmpower Framework Benchmarks: Round 20

89 points by pella 6y ago

jsnk 6y ago
I love looking at these benchmarks. Thank you TechEmpower for the work.
jsnk 6y ago
I was surpised to see javascript framework called, just-js show up at rank 9. In case, people are wondering, it looks like a new V8 runtime environment. https://github.com/just-js/just https://github.com/just-js/just
- flockonus 6y ago
  More info here: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01...
- matt42 6y ago
  To me there is 3 main points: - Even if it's JS, just-js make almost no use of dynamic memory allocations. - The authors rewrote himself a postgresql driver that support batching request. - it wraps high performance c++ libraries Even if the techempower implementation is far more verbose and complex than mainstream JS framework, the amount of work behind just-js and its preformances are just impressive!
- Volundr 6y ago
  I find myself a bit annoyed that they are calling that a "realistic" approach rather than stripped. That doesn't look like any "realistic" javascript I've ever seen in the wild: https://github.com/TechEmpower/FrameworkBenchmarks/tree/0096f84ee91ac9e7e2875a10e3a706d38069e8d7/frameworks/JavaScript/just https://github.com/TechEmpower/FrameworkBenchmarks/tree/0096...
pella 6y ago
github repo ( if any framework is missing .. ) - https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- lstamour 6y ago
  Note also, another reason to browse GitHub is to look at how the various implementations are written. For example, sometimes frameworks use fewer SQL statements to complete the same work, or they configure database libraries with different settings. I’ve noted a few instances where gaming these benchmarks is possible simply because a new framework implements the algorithm differently. Not to say they don’t have value — they do — but keep in mind your code may look very different from benchmark code. I do think it’s worth looking at the code to examine why some approaches scale better than others, though! Rarely is it the case that more code = faster performance, but sometimes features you need are sacrificed for extra performance even when a benchmark is classified as “realistic”. Viewing source code is how you can determine which frameworks actually best fit your needs, rather than just which ones performed well in this test… For example, it’s easy to think that raw Go performance is terrible compared to Rust or C++, but it’s worth pointing out that the raw Go example uses commonly available standard libraries baked in to the language to accomplish almost everything it does. Most other examples at some level or another have to deal with epoll and concurrency manually themselves or via the web server replacements they use. Based on this, the overhead or raw implementation of socket management is often what is being benchmarked. Implementations like just-js use C++ and epoll under the hood to implement their server, like most of the other high performance examples at the top of the benchmarks. The question often then becomes, do you want to write your own HTTP server or take advantage of an API someone else has written that might have more features or middleware? And remember that configuration matters, I forget off the top of my head, but the Go examples with a new web server had more performance optimization than the raw-Go benchmark had. Obviously I should submit a PR to fix this, but my ultimate goal here is to say that reading the source code will both help you become a better developer but also catch the shortcuts and techniques used by high performance implementations…
  - oaiey 6y ago
zinclozenge 6y ago
The original author of actix has a new framework called ntex. Looks like it was a fork of actix and then went from there. One of the differences I was able to spot was that it appears that it uses a (radix?) trie for the router.
- gohbgl 6y ago
  A couple of frameworks use a radix tree for routing, e.g. ASP .NET Core. Sadly, routing isn't even part of the TE benchmarks. There are some applications out there with 500+ routes, so I think it is pretty relevant.
da_big_ghey 6y ago
Random question: why is rocket so low in the rankings? Especially since something like actix is very high.
- swsieber 6y ago
  The latest stable rocket release doesn't use async, actix does. The master branch of rocket does have async (and stable rust, not nightly!) support.
  - yen223 6y ago
    If there's one lesson to draw from benchmarks like these, it is how important non-blocking IO is to high-performance web services
    - lmm 6y ago
      It's super important for highly concurrent, low-compute web services. Whether that's actually your business requirement is very situational.
- Freaky 6y ago
  Probably the lack of HTTP keep-alive support - it's explicitly disabled due to bugs in the older HTTP library the stable version uses (seen as a large number of errors in previous rankings), so each request needs a new TCP connection. https://github.com/TechEmpower/FrameworkBenchmarks/blob/f34037cef59727c88939b6b690704ac213443a67/frameworks/Rust/rocket/src/main.rs#L201 https://github.com/TechEmpower/FrameworkBenchmarks/blob/f340...
hkarthik 6y ago
I see a lot less value in these benchmarks than I did 6 years ago or so when they first started appearing. Maybe I'm showing my age but the speed argument just isn't that important to me any more. These days when I'm evaluating a framework, I look for strong documentation, integration with tools for observability and deployment, and compatibility with a well understood runtime that makes operations easier. These things are way more important for building software that works, that scales, and is operationally efficient for a team of engineers to work with for several years. Everything else is just vanity metrics.
- blunte 6y ago
  It would be very interesting to see an additional column of "prime examples" which link to the biggest or most heavily used web apps/apis built on a given stack. I suspect the examples would be few at the top of the list and many near the bottom.
- idoubtit 6y ago
  Obviously, performance generally isn't the main criteria when choosing a framework. I would argue that your "software that scales" isn't important either in most cases. I don't care much about performance either, especially compared to documentation, but that does not mean these benchmarks are useless. For instance, comparing similar frameworks can show big differences which illustrate different software architectures. Among the Nginx/phpfpm/Mysql fullstack-with-ORM frameworks, Laravel has roughly 8% of the raw stack capacity, while the other mature frameworks can do 2× or 3× more. I suspect this reflects Laravel's abuse of magic (calls to non-existent methods are redirected at runtime to other objects created on-the-fly, and so on). This may not be the real cause, but such a poor performance makes me suspicious about the complexity or the quality.
- lstamour 6y ago
  To me the interesting part is when you can read the benchmark source code and learn about a high performance feature for the framework or language you’re already using. That’s happened a few times for me with Go, for example. I hope more high performance enhancements are added even for frameworks that would appear near the bottom. That’s what keeps this relevant to me. :) Well, that and knowing roughly what the performance ceiling is for raw socket HTTP in various languages ;-)
pella 6y ago
Some framework is using extreme optimization tricks, so be careful .. and check the source code! :) example: "-d:danger : Turns off all runtime checks and turns on the optimizer." https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Nim/httpbeast/httpbeast.dockerfile#L12 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
- capableweb 6y ago
  > Some framework is using extreme optimization tricks, so be careful That accurately represents how I've been forced to run applications in production too, when under huge stress and we're trying to squeeze everything out of the existing hardware. Welcome to _Hacker_ News ;) Edit: And absolutely yes, check out the code behind it. Every benchmark has their flaws and biases so it's important to verify before making conclusions.
- dom96 6y ago
  I wouldn’t call this extreme, Nim is just honest about its optimisation flags :)
- deleted 6y ago
  [deleted]
lstamour 6y ago
I wrote this comment based on pre-release benchmarks, based on Physical benchmarks run against Git commits from Jan 23 - but it should still largely apply to the final benchmarks: It’s kind of frustrating to spend a few hours investigating preview results of the latest techempower benchmarks just to come to the conclusion that (a) truly high performance requires tuning your code to linux, to the network interface cards, and a deep understanding of what your code is doing and (b) right now at the top of the benchmarks, common high performance Rust code is mostly “unsafe” memory-wise, and the same is true of C++ for obvious reasons. After that: * When looking at garbage collected languages with very minimal implementations (Jooby mostly), Java can come out on top but requires 4 Gigs of memory which is more than I would ever like to use. * After Java comes C# but performance slows down the second you want to do anything less optimized, like use third-party libraries. * Finally, we’ve Go, which in a surprising twist is basically neck and neck with some pure PHP server implementations, but once I include those, I might as well mention there’s a JS implementation that skips the Node.js async event loop and offers performance on par with C++ and Rust, but heavily uses C++ internally and is thus more “unsafe”. What this tells me is kind of what I expected, implementation matters more than language… at this point, memory usage aside for Java, an efficient HTTP/1.1 server can be implemented in basically any language and when tuned or stripped down tends to run faster than the commonly used web servers, like Nginx, Tomcat, Express, etc. Often this means writing plain text RAW HTTP to a socket and managing sockets efficiently. Which brings us back to Go, though. Of all the implementations I’ve looked at so far, even the PHP one, only the Go benchmark is written like “standard Go” such that most people writing a service in Go will write something high performance without trying to make it high performance. Effectively, you don’t need to ensure all your dependencies use special optimized code routines to get something relatively optimized working quickly in Go with low memory overhead unlike Java. I’m a bit shocked by this conclusion as I was really hoping that Rust’s high performance use cases would win out, as it’s true that Rust can get 3x faster than Go, but on the Rust side, both actix and ntex are too immature as neither’s hit 1.0 yet, while tokio has hit 1.0 but its server, warp, is slower than Go. Irritating is that each of the techempower benchmarks use different implementations. For example, there’s a benchmark that’s supposed to measure 20 calls to Postgres, but one benchmark gets to the top by making only one long SQL statement that changes 20 rows. Another implementation uses pgx’s Batch functionality to send multiple queries in batches (a big timesaver, but not technically standard libpq), but then the standard Go variant doesn’t use Batching even when it could (which means we can’t compare custom implementations to generic Go ones fairly): https://github.com/TechEmpower/FrameworkBenchmarks/search?l=Go&q=batch https://github.com/TechEmpower/FrameworkBenchmarks/search?l=...
blunte 6y ago
These always make me feel bad. Everything I use or have used is in the < 15% performance rank. Do the ones at the top (like Actix Pg for example) provide everything you need to do real development, or are they stripped down? In other words, is this comparing track bikes to cross country bikes?
- lstamour 6y ago
  If you read the source code, it’s fair to say that while you can use the frameworks and implementations at the top to build your own systems, they would likely perform slower than these benchmarks as you add more features to them. Now, you might not need to add features, and that’s fair, but then you should compare implementations equally and that’s impossible when not every benchmark is written with identical algorithms and implementation code, for, well, obvious reasons. It’s worth looking at low level HTTP, socket and concurrency management if you want faster performance, but that’s not really a language-exclusive feature at that point. And the more realistic you make the benchmark — the more communication between microservices, for example - the more your application architecture, deployment hardware, kernel and network tuning, and so on can play a role. I am reminded of http://rachelbythebay.com/w/2020/10/14/lag/ http://rachelbythebay.com/w/2020/10/14/lag/ for example, as something really low level you probably don’t need to worry about… until you do. http://rachelbythebay.com/w/2020/05/07/serv/ http://rachelbythebay.com/w/2020/05/07/serv/ also. The same is true of most of the performance optimization at the top. Really fast? Yes. Useful? Depends on the rest of your code. Are you writing haproxy? How much does your app really need to do? Read the benchmark source code and get inspired, maybe. My conclusions for the moment: Switch to Go if low memory usage and high performance is as critical to you as post-1.0 stability. Go is generally stable these days ;-) Otherwise if instability can be tolerated but you want high performance, use one of these newer web server techniques and write a web server in unmanaged C++ that has very minimal functionality with language bindings. Just-js can serve as an example. Heck, the PHP benchmarks show that if you use PHP to write your own HTTP server you can still achieve high performance. That tells me the advantage here goes to web servers that literally “do less” rather than picking a language as faster over another language… especially when most (all?) languages can interface with C or C++ to do the HTTP layer at high performance while writing your code in whatever language you choose…
open-source-ux 6y ago
Some random observations... PHP Impressed by PHP's performance - it makes an entrance at position 20. If you exclude Javascript, the next interpreted language is Lua at position 81, then Python all the way down at 206. Ruby makes its first appearance at 255. Julia and Nim These are both new, modern languages that tout their performance as a benefit. It's a shame they did not take part in the benchmarks. *** Regardless of what you think of the benchmarks, the rankings do affect people's perceptions of languages and frameworks (both positive and negative). For example, I don't use PHP, but these benchmarks tell me that PHP has leapfrogged over Python and Ruby in the performance stakes for web development. And not just by a small margin, but by a significant difference.
- Xevi 6y ago
  There are 3 Nim frameworks in the plaintext benchmark. Httpbeast, Jester and Prologue.
  - jasfi 6y ago
    I did a quick check and only found them under the JSON serialization test. So they are missing from most tests.
- mdasen 6y ago
  I like the TechEmpower benchmarks, but you have to read the code to really understand the comparisons being made. In many languages, a bunch of web server code might end up being implemented in C and not the language itself. When you're creating a minimal endpoint for a benchmark, you might be exercising that C code and not the language itself. For example, the PHP implementation uses `htmlspecialchars` to encode the information which is implemented in C. That doesn't tell you much about the performance of PHP, but just that it has an optimized HTML escaper. The `asort` function used to sort the results is implemented in C and has more limited functionality compared to more general sorting functions that might be able to take lambdas in other languages. The PHP implementation even takes advantage of `PDO::FETCH_KEY_PAIR` which will only work if there are only two columns. Likewise, the PHP implementation doesn't use templates, but manually builds strings. The fastest Python implementation actually renders a Jinja2 template: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/uvicorn/app.py#L150 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast.... That's much more realistic in terms of what you'd do in the real world, but you're going to be carrying the overhead of a real-world template system like Jinja2. Part of Python's failure here is that no one wanted to implement an optimized Python version that would just build a string instead of rendering a template. Changing the test constraints a bit would ruin a lot of the advantages that PHP used there. Let's say that you had to retrieve three columns: `id, sort_key, fortune_text` and sort on the `sort_key`. Now you need to read more information back rather than just being able to make it an associative array (hash map). You need to be able to sort based on that sort_key which means probably giving a sort call a lambda. This isn't limited to PHP. A bunch of Go implementations do things like allocating a pool of structs and then re-using the same structs to avoid the garbage collector. A lot of implementations create result arrays sized so that they won't need to be re-sized (creating additional allocations and additional GC work). The rules say this isn't allowed, but they do it anyway. So, before comparing tests, I'd look at the implementations to make sure that they're comparable. A Django implementation that actually returns objects and is rendering templates and looks like a canonical Django implementation is very different from an optimized PHP version trying to avoid running any PHP as much as possible. When we start looking at the popular PHP frameworks which will be executing a bit of PHP like CodeIgniter or Laravel, we start seeing performance similar to Python frameworks as the PHP code is doing similar things like rendering templates. It just happens that no one implemented a Python version that didn't use a fully-fledged template renderer. And this is the weird thing: the benchmarks changed your perception of PHP while comparing things that weren't similar. I think PHP is often faster than Python and Ruby, but probably not to the extent that your perception might be given these benchmarks. I actually find it fascinating to look at the implementations and see which communities care about realistic implementations vs. leveraging all sorts of tricks to win the benchmark.
tomcam 6y ago
Mad props to these guys for keeping this massive project going for so long. Seems like a massive amount of work only to get picked apart by my friends here on HN.
- bhauer 6y ago
  Thanks, tomcam. It is a lot of work, but it's what we do in between our paying work. And we find it fascinating and fun. I think what really hits home are the stories about people upgrading elements at the foundation of their stack—their platform or framework—after those foundations have seen optimization efforts. It is rewarding to see application developers realize dramatic performance boosts within their applications with so little effort; we feel we have contributed to this delight in a small way. For example, check out this post by the Azure Active Directory team [1] where they saw massive performance improvements by upgrading from .NET Framework to .NET Core 3.1 (which isn't even the latest version). Some people focus too much on specific rank ordering and the optimization efforts made by those who jockey for top spots. It's better to consume the data in rough tiers of performance, however you choose to define those tiers. We tend to encourage people to consider performance as one part of the puzzle when selecting infrastructure software. Several high performance platforms and frameworks (in Java, Rust, C#, Go, Python, and more) are also ergonomic and easy to work with. And using high performance infrastructure software means you can avoid premature optimization in your application code and avoid premature architectual complexity. You can enjoy the trifecta of architectural simplicity, low-latency, and good scale headroom. Obviously we know this project will always receive diverse and critical opinions. That's fine; hackers are a very opinionated people. For those who value the data, ourselves included, we are happy to keep putting the effort in. [1] https://devblogs.microsoft.com/dotnet/azure-active-directorys-gateway-service-is-on-net-core-3-1/ https://devblogs.microsoft.com/dotnet/azure-active-directory...
  - billywhizz 6y ago
    i think it is really interesting to see the changes in different languages and platforms over time and also very useful to the community to see the optimisations folks come up with. thanks for the continued work brian and team!
noncoml 6y ago
The way I like to look at this data is in the "Multiple queries" test and sorted by "Max latency"
jakearmitage 6y ago
The top PHP entry, "php-ngx-pgsql", uses a quite obscure way of running PHP and Nginx: https://github.com/rryqszq4/ngx_php7 https://github.com/rryqszq4/ngx_php7 https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/PHP/php-ngx/deploy/nginx.conf#L49 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Very similar to OpenResty, but with PHP instead of Lua. Which is funny, because OpenResty ranks lower.
sprite 6y ago
These results really make me want to try out ASP.NET Core.
- HotVector 6y ago
  Code doesn't make sense, weird ass abstraction, but works beautifully. I never could understand where the dependency injection parameters came from lmao
philliphaydon 6y ago
I think some people are knocking these benchmarks for the wrong reason. I’m a .NET dev primarily. And .net has historically been a terrible performer on these benchmarks. With .net core the asp.net team and community has been working on performance. While the raw plain text benchmark is not really realistic of a real world scenario that we would use on a day to day basis. It is an indicator of the performance baseline that .net core can achieve before you begin adding all the fluff on top. This coupled with sites like stack overflow showing the gains they get using asp.net and moving from 1 version to the next gives me confidence that .net is not a bad choice these days. Prior to .net core I was more or less ready to go switch full time to something else. But now I’m happy using .net. /my 2c
- jbergens 6y ago
  Agree, that MS is working on performance and can show it on TechEmpower benchmarks is great. I also think some top performers are using special ways to be that fast but if you move down a bit in the list there are probably more realistic implementations. I also got a bit suprised over this other benchmark showing how fast C# can be: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/csharp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - ygra 6y ago
    Always look at the source. Some of them are very representative of high-performance C#, but others cheat a little. E.g. regex-redux just calls out to PCRE, although .NET 5's Regex would easily be competitive with Java as well.
ksec 6y ago
Normally I just click on Fortune Cookies, since I think that is the most representative of a typical Web Workload. And then Click on Filter, Full Stack Framework, And Full ORM only. Again, most people will be using some sort of Web Framework with Full ORM. I am surprised the top results is from PHP. For many years it has always been Java. And it is from a framework called ubiquity [1] which I have never heard of it before. There is Crystal and Lucky [2]. Which is exciting because they are getting very close to 1.0 Ruby Rails... is still at the bottom of the pack. [1] https://github.com/phpMv/ubiquity https://github.com/phpMv/ubiquity [2] https://www.luckyframework.org https://www.luckyframework.org
danielscrubs 6y ago
What is the most fun and shocking to me to browse the code https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks to see just how unergonomic and messy the code gets in something like this. 20 something files in 10 directories fiddled with config files isn’t uncommon. Of the bunch, by only looking at the code, which of them would you like to get dumped on you 2 weeks before a deadline? Which is the most ergonomic?
- slezyr 6y ago
  Yeah, the benchmark is no longer useful due to all the micro-optimizations in the code that never happen in the real world. Most of them check the Nth letter of the HTTP method to do routing, hard code the content-length, etc. This benchmark shows which language is the fastest to throw the hardcoded data into the socket. Meantime, Django[1] is the simplest and a real-world application. 1: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/django/hello/world/views.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

89 points by pella 6y ago

jsnk 6y ago

I love looking at these benchmarks. Thank you TechEmpower for the work.

jsnk 6y ago

I was surpised to see javascript framework called, just-js show up at rank 9. In case, people are wondering, it looks like a new V8 runtime environment. https://github.com/just-js/just https://github.com/just-js/just

flockonus 6y ago
More info here: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01...
matt42 6y ago
To me there is 3 main points: - Even if it's JS, just-js make almost no use of dynamic memory allocations. - The authors rewrote himself a postgresql driver that support batching request. - it wraps high performance c++ libraries Even if the techempower implementation is far more verbose and complex than mainstream JS framework, the amount of work behind just-js and its preformances are just impressive!
Volundr 6y ago
I find myself a bit annoyed that they are calling that a "realistic" approach rather than stripped. That doesn't look like any "realistic" javascript I've ever seen in the wild: https://github.com/TechEmpower/FrameworkBenchmarks/tree/0096f84ee91ac9e7e2875a10e3a706d38069e8d7/frameworks/JavaScript/just https://github.com/TechEmpower/FrameworkBenchmarks/tree/0096...

pella 6y ago

github repo ( if any framework is missing .. ) - https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks

lstamour 6y ago
Note also, another reason to browse GitHub is to look at how the various implementations are written. For example, sometimes frameworks use fewer SQL statements to complete the same work, or they configure database libraries with different settings. I’ve noted a few instances where gaming these benchmarks is possible simply because a new framework implements the algorithm differently. Not to say they don’t have value — they do — but keep in mind your code may look very different from benchmark code. I do think it’s worth looking at the code to examine why some approaches scale better than others, though! Rarely is it the case that more code = faster performance, but sometimes features you need are sacrificed for extra performance even when a benchmark is classified as “realistic”. Viewing source code is how you can determine which frameworks actually best fit your needs, rather than just which ones performed well in this test… For example, it’s easy to think that raw Go performance is terrible compared to Rust or C++, but it’s worth pointing out that the raw Go example uses commonly available standard libraries baked in to the language to accomplish almost everything it does. Most other examples at some level or another have to deal with epoll and concurrency manually themselves or via the web server replacements they use. Based on this, the overhead or raw implementation of socket management is often what is being benchmarked. Implementations like just-js use C++ and epoll under the hood to implement their server, like most of the other high performance examples at the top of the benchmarks. The question often then becomes, do you want to write your own HTTP server or take advantage of an API someone else has written that might have more features or middleware? And remember that configuration matters, I forget off the top of my head, but the Go examples with a new web server had more performance optimization than the raw-Go benchmark had. Obviously I should submit a PR to fix this, but my ultimate goal here is to say that reading the source code will both help you become a better developer but also catch the shortcuts and techniques used by high performance implementations…
- oaiey 6y ago

zinclozenge 6y ago

The original author of actix has a new framework called ntex. Looks like it was a fork of actix and then went from there. One of the differences I was able to spot was that it appears that it uses a (radix?) trie for the router.

gohbgl 6y ago
A couple of frameworks use a radix tree for routing, e.g. ASP .NET Core. Sadly, routing isn't even part of the TE benchmarks. There are some applications out there with 500+ routes, so I think it is pretty relevant.

da_big_ghey 6y ago

Random question: why is rocket so low in the rankings? Especially since something like actix is very high.

swsieber 6y ago
The latest stable rocket release doesn't use async, actix does. The master branch of rocket does have async (and stable rust, not nightly!) support.
- yen223 6y ago
  If there's one lesson to draw from benchmarks like these, it is how important non-blocking IO is to high-performance web services
  - lmm 6y ago
    It's super important for highly concurrent, low-compute web services. Whether that's actually your business requirement is very situational.
Freaky 6y ago
Probably the lack of HTTP keep-alive support - it's explicitly disabled due to bugs in the older HTTP library the stable version uses (seen as a large number of errors in previous rankings), so each request needs a new TCP connection. https://github.com/TechEmpower/FrameworkBenchmarks/blob/f34037cef59727c88939b6b690704ac213443a67/frameworks/Rust/rocket/src/main.rs#L201 https://github.com/TechEmpower/FrameworkBenchmarks/blob/f340...

hkarthik 6y ago

I see a lot less value in these benchmarks than I did 6 years ago or so when they first started appearing. Maybe I'm showing my age but the speed argument just isn't that important to me any more. These days when I'm evaluating a framework, I look for strong documentation, integration with tools for observability and deployment, and compatibility with a well understood runtime that makes operations easier. These things are way more important for building software that works, that scales, and is operationally efficient for a team of engineers to work with for several years. Everything else is just vanity metrics.

blunte 6y ago
It would be very interesting to see an additional column of "prime examples" which link to the biggest or most heavily used web apps/apis built on a given stack. I suspect the examples would be few at the top of the list and many near the bottom.
idoubtit 6y ago
Obviously, performance generally isn't the main criteria when choosing a framework. I would argue that your "software that scales" isn't important either in most cases. I don't care much about performance either, especially compared to documentation, but that does not mean these benchmarks are useless. For instance, comparing similar frameworks can show big differences which illustrate different software architectures. Among the Nginx/phpfpm/Mysql fullstack-with-ORM frameworks, Laravel has roughly 8% of the raw stack capacity, while the other mature frameworks can do 2× or 3× more. I suspect this reflects Laravel's abuse of magic (calls to non-existent methods are redirected at runtime to other objects created on-the-fly, and so on). This may not be the real cause, but such a poor performance makes me suspicious about the complexity or the quality.
lstamour 6y ago
To me the interesting part is when you can read the benchmark source code and learn about a high performance feature for the framework or language you’re already using. That’s happened a few times for me with Go, for example. I hope more high performance enhancements are added even for frameworks that would appear near the bottom. That’s what keeps this relevant to me. :) Well, that and knowing roughly what the performance ceiling is for raw socket HTTP in various languages ;-)

pella 6y ago

Some framework is using extreme optimization tricks, so be careful .. and check the source code! :) example: "-d:danger : Turns off all runtime checks and turns on the optimizer." https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Nim/httpbeast/httpbeast.dockerfile#L12 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

capableweb 6y ago
> Some framework is using extreme optimization tricks, so be careful That accurately represents how I've been forced to run applications in production too, when under huge stress and we're trying to squeeze everything out of the existing hardware. Welcome to _Hacker_ News ;) Edit: And absolutely yes, check out the code behind it. Every benchmark has their flaws and biases so it's important to verify before making conclusions.
dom96 6y ago
I wouldn’t call this extreme, Nim is just honest about its optimisation flags :)
deleted 6y ago
[deleted]

lstamour 6y ago

I wrote this comment based on pre-release benchmarks, based on Physical benchmarks run against Git commits from Jan 23 - but it should still largely apply to the final benchmarks: It’s kind of frustrating to spend a few hours investigating preview results of the latest techempower benchmarks just to come to the conclusion that (a) truly high performance requires tuning your code to linux, to the network interface cards, and a deep understanding of what your code is doing and (b) right now at the top of the benchmarks, common high performance Rust code is mostly “unsafe” memory-wise, and the same is true of C++ for obvious reasons. After that: * When looking at garbage collected languages with very minimal implementations (Jooby mostly), Java can come out on top but requires 4 Gigs of memory which is more than I would ever like to use. * After Java comes C# but performance slows down the second you want to do anything less optimized, like use third-party libraries. * Finally, we’ve Go, which in a surprising twist is basically neck and neck with some pure PHP server implementations, but once I include those, I might as well mention there’s a JS implementation that skips the Node.js async event loop and offers performance on par with C++ and Rust, but heavily uses C++ internally and is thus more “unsafe”. What this tells me is kind of what I expected, implementation matters more than language… at this point, memory usage aside for Java, an efficient HTTP/1.1 server can be implemented in basically any language and when tuned or stripped down tends to run faster than the commonly used web servers, like Nginx, Tomcat, Express, etc. Often this means writing plain text RAW HTTP to a socket and managing sockets efficiently. Which brings us back to Go, though. Of all the implementations I’ve looked at so far, even the PHP one, only the Go benchmark is written like “standard Go” such that most people writing a service in Go will write something high performance without trying to make it high performance. Effectively, you don’t need to ensure all your dependencies use special optimized code routines to get something relatively optimized working quickly in Go with low memory overhead unlike Java. I’m a bit shocked by this conclusion as I was really hoping that Rust’s high performance use cases would win out, as it’s true that Rust can get 3x faster than Go, but on the Rust side, both actix and ntex are too immature as neither’s hit 1.0 yet, while tokio has hit 1.0 but its server, warp, is slower than Go. Irritating is that each of the techempower benchmarks use different implementations. For example, there’s a benchmark that’s supposed to measure 20 calls to Postgres, but one benchmark gets to the top by making only one long SQL statement that changes 20 rows. Another implementation uses pgx’s Batch functionality to send multiple queries in batches (a big timesaver, but not technically standard libpq), but then the standard Go variant doesn’t use Batching even when it could (which means we can’t compare custom implementations to generic Go ones fairly): https://github.com/TechEmpower/FrameworkBenchmarks/search?l=Go&q=batch https://github.com/TechEmpower/FrameworkBenchmarks/search?l=...

blunte 6y ago

These always make me feel bad. Everything I use or have used is in the < 15% performance rank. Do the ones at the top (like Actix Pg for example) provide everything you need to do real development, or are they stripped down? In other words, is this comparing track bikes to cross country bikes?

lstamour 6y ago
If you read the source code, it’s fair to say that while you can use the frameworks and implementations at the top to build your own systems, they would likely perform slower than these benchmarks as you add more features to them. Now, you might not need to add features, and that’s fair, but then you should compare implementations equally and that’s impossible when not every benchmark is written with identical algorithms and implementation code, for, well, obvious reasons. It’s worth looking at low level HTTP, socket and concurrency management if you want faster performance, but that’s not really a language-exclusive feature at that point. And the more realistic you make the benchmark — the more communication between microservices, for example - the more your application architecture, deployment hardware, kernel and network tuning, and so on can play a role. I am reminded of http://rachelbythebay.com/w/2020/10/14/lag/ http://rachelbythebay.com/w/2020/10/14/lag/ for example, as something really low level you probably don’t need to worry about… until you do. http://rachelbythebay.com/w/2020/05/07/serv/ http://rachelbythebay.com/w/2020/05/07/serv/ also. The same is true of most of the performance optimization at the top. Really fast? Yes. Useful? Depends on the rest of your code. Are you writing haproxy? How much does your app really need to do? Read the benchmark source code and get inspired, maybe. My conclusions for the moment: Switch to Go if low memory usage and high performance is as critical to you as post-1.0 stability. Go is generally stable these days ;-) Otherwise if instability can be tolerated but you want high performance, use one of these newer web server techniques and write a web server in unmanaged C++ that has very minimal functionality with language bindings. Just-js can serve as an example. Heck, the PHP benchmarks show that if you use PHP to write your own HTTP server you can still achieve high performance. That tells me the advantage here goes to web servers that literally “do less” rather than picking a language as faster over another language… especially when most (all?) languages can interface with C or C++ to do the HTTP layer at high performance while writing your code in whatever language you choose…

open-source-ux 6y ago

Some random observations... PHP Impressed by PHP's performance - it makes an entrance at position 20. If you exclude Javascript, the next interpreted language is Lua at position 81, then Python all the way down at 206. Ruby makes its first appearance at 255. Julia and Nim These are both new, modern languages that tout their performance as a benefit. It's a shame they did not take part in the benchmarks. *** Regardless of what you think of the benchmarks, the rankings do affect people's perceptions of languages and frameworks (both positive and negative). For example, I don't use PHP, but these benchmarks tell me that PHP has leapfrogged over Python and Ruby in the performance stakes for web development. And not just by a small margin, but by a significant difference.

Xevi 6y ago
There are 3 Nim frameworks in the plaintext benchmark. Httpbeast, Jester and Prologue.
- jasfi 6y ago
  I did a quick check and only found them under the JSON serialization test. So they are missing from most tests.
mdasen 6y ago
I like the TechEmpower benchmarks, but you have to read the code to really understand the comparisons being made. In many languages, a bunch of web server code might end up being implemented in C and not the language itself. When you're creating a minimal endpoint for a benchmark, you might be exercising that C code and not the language itself. For example, the PHP implementation uses `htmlspecialchars` to encode the information which is implemented in C. That doesn't tell you much about the performance of PHP, but just that it has an optimized HTML escaper. The `asort` function used to sort the results is implemented in C and has more limited functionality compared to more general sorting functions that might be able to take lambdas in other languages. The PHP implementation even takes advantage of `PDO::FETCH_KEY_PAIR` which will only work if there are only two columns. Likewise, the PHP implementation doesn't use templates, but manually builds strings. The fastest Python implementation actually renders a Jinja2 template: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/uvicorn/app.py#L150 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast.... That's much more realistic in terms of what you'd do in the real world, but you're going to be carrying the overhead of a real-world template system like Jinja2. Part of Python's failure here is that no one wanted to implement an optimized Python version that would just build a string instead of rendering a template. Changing the test constraints a bit would ruin a lot of the advantages that PHP used there. Let's say that you had to retrieve three columns: `id, sort_key, fortune_text` and sort on the `sort_key`. Now you need to read more information back rather than just being able to make it an associative array (hash map). You need to be able to sort based on that sort_key which means probably giving a sort call a lambda. This isn't limited to PHP. A bunch of Go implementations do things like allocating a pool of structs and then re-using the same structs to avoid the garbage collector. A lot of implementations create result arrays sized so that they won't need to be re-sized (creating additional allocations and additional GC work). The rules say this isn't allowed, but they do it anyway. So, before comparing tests, I'd look at the implementations to make sure that they're comparable. A Django implementation that actually returns objects and is rendering templates and looks like a canonical Django implementation is very different from an optimized PHP version trying to avoid running any PHP as much as possible. When we start looking at the popular PHP frameworks which will be executing a bit of PHP like CodeIgniter or Laravel, we start seeing performance similar to Python frameworks as the PHP code is doing similar things like rendering templates. It just happens that no one implemented a Python version that didn't use a fully-fledged template renderer. And this is the weird thing: the benchmarks changed your perception of PHP while comparing things that weren't similar. I think PHP is often faster than Python and Ruby, but probably not to the extent that your perception might be given these benchmarks. I actually find it fascinating to look at the implementations and see which communities care about realistic implementations vs. leveraging all sorts of tricks to win the benchmark.

tomcam 6y ago

Mad props to these guys for keeping this massive project going for so long. Seems like a massive amount of work only to get picked apart by my friends here on HN.

bhauer 6y ago
Thanks, tomcam. It is a lot of work, but it's what we do in between our paying work. And we find it fascinating and fun. I think what really hits home are the stories about people upgrading elements at the foundation of their stack—their platform or framework—after those foundations have seen optimization efforts. It is rewarding to see application developers realize dramatic performance boosts within their applications with so little effort; we feel we have contributed to this delight in a small way. For example, check out this post by the Azure Active Directory team [1] where they saw massive performance improvements by upgrading from .NET Framework to .NET Core 3.1 (which isn't even the latest version). Some people focus too much on specific rank ordering and the optimization efforts made by those who jockey for top spots. It's better to consume the data in rough tiers of performance, however you choose to define those tiers. We tend to encourage people to consider performance as one part of the puzzle when selecting infrastructure software. Several high performance platforms and frameworks (in Java, Rust, C#, Go, Python, and more) are also ergonomic and easy to work with. And using high performance infrastructure software means you can avoid premature optimization in your application code and avoid premature architectual complexity. You can enjoy the trifecta of architectural simplicity, low-latency, and good scale headroom. Obviously we know this project will always receive diverse and critical opinions. That's fine; hackers are a very opinionated people. For those who value the data, ourselves included, we are happy to keep putting the effort in. [1] https://devblogs.microsoft.com/dotnet/azure-active-directorys-gateway-service-is-on-net-core-3-1/ https://devblogs.microsoft.com/dotnet/azure-active-directory...
- billywhizz 6y ago
  i think it is really interesting to see the changes in different languages and platforms over time and also very useful to the community to see the optimisations folks come up with. thanks for the continued work brian and team!

noncoml 6y ago

The way I like to look at this data is in the "Multiple queries" test and sorted by "Max latency"

jakearmitage 6y ago

The top PHP entry, "php-ngx-pgsql", uses a quite obscure way of running PHP and Nginx: https://github.com/rryqszq4/ngx_php7 https://github.com/rryqszq4/ngx_php7 https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/PHP/php-ngx/deploy/nginx.conf#L49 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Very similar to OpenResty, but with PHP instead of Lua. Which is funny, because OpenResty ranks lower.

sprite 6y ago

These results really make me want to try out ASP.NET Core.

HotVector 6y ago
Code doesn't make sense, weird ass abstraction, but works beautifully. I never could understand where the dependency injection parameters came from lmao

philliphaydon 6y ago

I think some people are knocking these benchmarks for the wrong reason. I’m a .NET dev primarily. And .net has historically been a terrible performer on these benchmarks. With .net core the asp.net team and community has been working on performance. While the raw plain text benchmark is not really realistic of a real world scenario that we would use on a day to day basis. It is an indicator of the performance baseline that .net core can achieve before you begin adding all the fluff on top. This coupled with sites like stack overflow showing the gains they get using asp.net and moving from 1 version to the next gives me confidence that .net is not a bad choice these days. Prior to .net core I was more or less ready to go switch full time to something else. But now I’m happy using .net. /my 2c

jbergens 6y ago
Agree, that MS is working on performance and can show it on TechEmpower benchmarks is great. I also think some top performers are using special ways to be that fast but if you move down a bit in the list there are probably more realistic implementations. I also got a bit suprised over this other benchmark showing how fast C# can be: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/csharp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- ygra 6y ago
  Always look at the source. Some of them are very representative of high-performance C#, but others cheat a little. E.g. regex-redux just calls out to PCRE, although .NET 5's Regex would easily be competitive with Java as well.

ksec 6y ago

Normally I just click on Fortune Cookies, since I think that is the most representative of a typical Web Workload. And then Click on Filter, Full Stack Framework, And Full ORM only. Again, most people will be using some sort of Web Framework with Full ORM. I am surprised the top results is from PHP. For many years it has always been Java. And it is from a framework called ubiquity [1] which I have never heard of it before. There is Crystal and Lucky [2]. Which is exciting because they are getting very close to 1.0 Ruby Rails... is still at the bottom of the pack. [1] https://github.com/phpMv/ubiquity https://github.com/phpMv/ubiquity [2] https://www.luckyframework.org https://www.luckyframework.org

danielscrubs 6y ago

What is the most fun and shocking to me to browse the code https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks to see just how unergonomic and messy the code gets in something like this. 20 something files in 10 directories fiddled with config files isn’t uncommon. Of the bunch, by only looking at the code, which of them would you like to get dumped on you 2 weeks before a deadline? Which is the most ergonomic?

slezyr 6y ago
Yeah, the benchmark is no longer useful due to all the micro-optimizations in the code that never happen in the real world. Most of them check the Nth letter of the HTTP method to do routing, hard code the content-length, etc. This benchmark shows which language is the fastest to throw the hardcoded data into the socket. Meantime, Django[1] is the simplest and a real-world application. 1: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/django/hello/world/views.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...