4 ms·

Is this a criteria that's important enough to warrant this much effort? I can think of several metrics that would come first - even for sites with fairly high

Is this a criteria that's important enough to warrant this much effort? I can think of several metrics that would come first - even for sites with fairly high performance requirements. How often is the web framework genuinely the bottleneck (at least in a way that's not easily fixable with caching or any other similar sledgehammer)?

matt42 6y ago
The benchmark show that the frameworks has a big impact on performance. While all the frameworks are using the same database server, performances varies from 1x to 100x. While the benchmark focus on more req/s, we can also see it the other way around: for the same req/s, you can use smaller hardware, i.e. save money and energy.
- ghostwriter 6y ago
  Not every framework presented in the benchmark perform the same amount of necessary protocol verifications. For instance, here's the take from Servant perspective - https://www.reddit.com/r/haskell/comments/6b3dlt/techempower_benchmarks_14_released_with_servant/dhl500h/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/haskell/comments/6b3dlt/techempower...
stagas 6y ago
In a typical web app you probably wouldn't notice any difference but in the niche where you're handling millions of reqs/s even slight improvements there could possibly translate to a non-trivial amount of money savings. On the other hand if you've already scaled to such a degree then those savings wouldn't matter that much. So, what's left is maybe for educational or research purposes, it's good to have an idea where the upper bound limit is while still maintaining a decent API, so there's value there, it's a great feat regardless.
- weego 6y ago
  At millions of reqs per second the thin contract of the internals of http request/response will never be the layer that will have any fruit left to pick to make gains.
  - sulam 6y ago
    You’re making a statement here that is only true if you assume the developers of such a service have optimized their framework. If instead they are using one of the slower frameworks, they will pay proportionally — you can’t assume that just because they take Mrps that the service will be super complex past http. Often the exact opposite is true and they get to that scale by keeping the service very, very simple. Http processing becomes -more- of their latency budget in such cases.
  - orthoxerox 6y ago
    Yes, and the top frameworks are already bandwidth limited on the simplest benchmarks. The big gains are now made in DB access, which is a much more complicated topic than "ORM is bad for performance".
rishav_sharan 6y ago
4/6 tests in there account for database requests. So yes, I would say that Tech empower is pretty important to gauge the performance and efficiency of web frameworks. Personally, it helped me realize that my mistaken belief that nodejs was hella fast and that old Java based frameworks are slow. It also introduced me to Kotlin (I started on it to try Kooby) and overall love that I can at a glance see that for my chosen set of languages, which farmeworks-dbs are a good bet.
- mattmanser 6y ago
  The greatest trick that javascript ever pulled was to convince the latest generation of programmers that it was fast...
- dathinab 6y ago
  The problem is they account for _simple_ database requests. Which surely are common but long(er) running database requests or delegation to another service aren't super rare either. Similar some of the frameworks might have major reliability problems which can randomly "crop up" due to thread pinning and similar. I believe it should have some form of "chaos" test where instead of interacting with the DB it interacts with a another test server on both ends and the front and back connected server run through long random "chaos" simulation (and another version where a pre-recorded run is used for reproducibility).
Alex3917 6y ago
Django is already vastly faster than human perception. An entire roundtrip to my Django app takes 70 - 110ms. The entire roundtrip is already faster than human perception even in the worst case, and of that Django itself probably contributes something like 2 or 3ms.
- ghostwriter 6y ago
  At this point it's mostly about (cost) efficiency, like how many EC2 instances you will need to cover your workload with a service implemented in a particular framework. It also assumes that developers can pick any of them with the same amount of effort, which is probably a proper assumption for a benchmark that measures performance.
  - Alex3917 6y ago
    How much cost efficiency is there? I’ve worked at multiple of the top 100 websites in the U.S., and none of them had more than a dozen web servers. For any website, 90% of traffic is logged out and probably doesn’t even hit the web servers. And web servers are generally the least expensive component of hosting costs.
    - ghostwriter 6y ago
      These costs are more meaningfull for those who bootstrap their businesses and projects at their own expense. I have a personal anectdote where switching from a scripting web framework to a compiled one reduced the runtime size, and the deployable environment went from 800mb to 10mb, which allowed me to fit all my deployments into a free-tier monthly quota of AWS ECR. Given I already knew both frameworks from fulltime jobs, it's a great net benefit. It also means that whoever wants to use self-hosted open-source alternatives of user-tracking services (notes, calendars, emails, etc), they can do it cheaper or even for free.
- ksec 6y ago
  >and of that Django itself probably contributes something like 2 or 3ms. Unless it is super tuned, running on a specialise VM, I seriously have doubt about that. It is already pretty damn good if PHP/Python Django and Ruby Rails could do 10ms request excluding the Database on a moderately large App. Generally they are 20ms+. And recently shopify even have their performance matrix allowing for 300ms per request.
austincheney 6y ago
On the client side, at least now, the framework and/or means of API access are always the primary performance bottleneck.
brightball 6y ago
It's also a question of what's going on when you dive in. Out of curiosity, I've looked into several parts of benchmark source code before and it's questionable especially where the database is concerned. You'll see some things that seem to go against their own rules, like presorting and batching inserts for efficiency but their rules seem to indicate that it's not allowed. It has lead me to treat these benchmarks with a grain of salt. There are very few circumstances that I can imagine that would send me running for C++ for a web project. I'm much more concerned about balancing efficiency of the developer environment with performance, security and maintainability, than pure performance. At the point that I'm not, my priorities are out of whack (aside from very niche situations). The side effect of these benchmarks, while we all love benchmarks, is the idea that any of these faster frameworks are actually a better choice because of a benchmark result.
- Areading314 6y ago
  There are reasons why you might need to bolt an http api into a c++ project, so it's nice to know there are high performance options available
- matt42 6y ago
  My motivation in writing yet another web framework was to get pure performance while providing maintainability of more dynamic languages. Check the implementation of lithium, it is one of the most concise implementation of TFB: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/lithium/lithium.cc https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

matt42 6y ago

The benchmark show that the frameworks has a big impact on performance. While all the frameworks are using the same database server, performances varies from 1x to 100x. While the benchmark focus on more req/s, we can also see it the other way around: for the same req/s, you can use smaller hardware, i.e. save money and energy.

ghostwriter 6y ago
Not every framework presented in the benchmark perform the same amount of necessary protocol verifications. For instance, here's the take from Servant perspective - https://www.reddit.com/r/haskell/comments/6b3dlt/techempower_benchmarks_14_released_with_servant/dhl500h/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/haskell/comments/6b3dlt/techempower...

stagas 6y ago

In a typical web app you probably wouldn't notice any difference but in the niche where you're handling millions of reqs/s even slight improvements there could possibly translate to a non-trivial amount of money savings. On the other hand if you've already scaled to such a degree then those savings wouldn't matter that much. So, what's left is maybe for educational or research purposes, it's good to have an idea where the upper bound limit is while still maintaining a decent API, so there's value there, it's a great feat regardless.

weego 6y ago
At millions of reqs per second the thin contract of the internals of http request/response will never be the layer that will have any fruit left to pick to make gains.
- sulam 6y ago
  You’re making a statement here that is only true if you assume the developers of such a service have optimized their framework. If instead they are using one of the slower frameworks, they will pay proportionally — you can’t assume that just because they take Mrps that the service will be super complex past http. Often the exact opposite is true and they get to that scale by keeping the service very, very simple. Http processing becomes -more- of their latency budget in such cases.
- orthoxerox 6y ago
  Yes, and the top frameworks are already bandwidth limited on the simplest benchmarks. The big gains are now made in DB access, which is a much more complicated topic than "ORM is bad for performance".

rishav_sharan 6y ago

4/6 tests in there account for database requests. So yes, I would say that Tech empower is pretty important to gauge the performance and efficiency of web frameworks. Personally, it helped me realize that my mistaken belief that nodejs was hella fast and that old Java based frameworks are slow. It also introduced me to Kotlin (I started on it to try Kooby) and overall love that I can at a glance see that for my chosen set of languages, which farmeworks-dbs are a good bet.

mattmanser 6y ago
The greatest trick that javascript ever pulled was to convince the latest generation of programmers that it was fast...
dathinab 6y ago
The problem is they account for _simple_ database requests. Which surely are common but long(er) running database requests or delegation to another service aren't super rare either. Similar some of the frameworks might have major reliability problems which can randomly "crop up" due to thread pinning and similar. I believe it should have some form of "chaos" test where instead of interacting with the DB it interacts with a another test server on both ends and the front and back connected server run through long random "chaos" simulation (and another version where a pre-recorded run is used for reproducibility).

Alex3917 6y ago

Django is already vastly faster than human perception. An entire roundtrip to my Django app takes 70 - 110ms. The entire roundtrip is already faster than human perception even in the worst case, and of that Django itself probably contributes something like 2 or 3ms.

ghostwriter 6y ago
At this point it's mostly about (cost) efficiency, like how many EC2 instances you will need to cover your workload with a service implemented in a particular framework. It also assumes that developers can pick any of them with the same amount of effort, which is probably a proper assumption for a benchmark that measures performance.
- Alex3917 6y ago
  How much cost efficiency is there? I’ve worked at multiple of the top 100 websites in the U.S., and none of them had more than a dozen web servers. For any website, 90% of traffic is logged out and probably doesn’t even hit the web servers. And web servers are generally the least expensive component of hosting costs.
  - ghostwriter 6y ago
    These costs are more meaningfull for those who bootstrap their businesses and projects at their own expense. I have a personal anectdote where switching from a scripting web framework to a compiled one reduced the runtime size, and the deployable environment went from 800mb to 10mb, which allowed me to fit all my deployments into a free-tier monthly quota of AWS ECR. Given I already knew both frameworks from fulltime jobs, it's a great net benefit. It also means that whoever wants to use self-hosted open-source alternatives of user-tracking services (notes, calendars, emails, etc), they can do it cheaper or even for free.
ksec 6y ago
>and of that Django itself probably contributes something like 2 or 3ms. Unless it is super tuned, running on a specialise VM, I seriously have doubt about that. It is already pretty damn good if PHP/Python Django and Ruby Rails could do 10ms request excluding the Database on a moderately large App. Generally they are 20ms+. And recently shopify even have their performance matrix allowing for 300ms per request.

austincheney 6y ago

On the client side, at least now, the framework and/or means of API access are always the primary performance bottleneck.

brightball 6y ago

It's also a question of what's going on when you dive in. Out of curiosity, I've looked into several parts of benchmark source code before and it's questionable especially where the database is concerned. You'll see some things that seem to go against their own rules, like presorting and batching inserts for efficiency but their rules seem to indicate that it's not allowed. It has lead me to treat these benchmarks with a grain of salt. There are very few circumstances that I can imagine that would send me running for C++ for a web project. I'm much more concerned about balancing efficiency of the developer environment with performance, security and maintainability, than pure performance. At the point that I'm not, my priorities are out of whack (aside from very niche situations). The side effect of these benchmarks, while we all love benchmarks, is the idea that any of these faster frameworks are actually a better choice because of a benchmark result.

Areading314 6y ago
There are reasons why you might need to bolt an http api into a c++ project, so it's nice to know there are high performance options available
matt42 6y ago
My motivation in writing yet another web framework was to get pure performance while providing maintainability of more dynamic languages. Check the implementation of lithium, it is one of the most concise implementation of TFB: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/lithium/lithium.cc https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...