5 ms·

I heard some old guard person at Mozilla talk about these benchmarks last year when Rust's ACTIX was on top, something like "whatever is the latest framework ca

I heard some old guard person at Mozilla talk about these benchmarks last year when Rust's ACTIX was on top, something like "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest", at the time I took it to mean ACTIX being the fastest has more to do with it being new then it being written in rust. Given a newcomer is at the top again that seems to ring true? Does seem odd older frameworks wouldn't keep up with that game, but perhaps once they get the publicity and user base they can worry more about features than having the highest benchmark?

ori_b 6y ago
People ask for more features over time. I'd expect older frameworks to be more featureful, but slower.
Klonoar 6y ago
It's worth noting that actix-web is at the top by doing some insane tricks, which you wouldn't normally do in writing a web application. e.g at one point they used Askama as their templating engine (may have switched), which isn't really fair as using Askama means you force a recompile of your project for each template change - and I can tell you from experience that your compile times will shoot through the roof as well. Nobody is going to bother doing that. I say all of this as an actix-web user. It's my go-to. I just view these benchmarks as "if I ever needed to, I know I can wrangle performance out of it... but I won't assume it by default".
- terhechte 6y ago
  But most other projects on there do that as well. I regard tech empower as the maximum performance I can get with a given framework if need be. With actin I can add Askama, with Vapor, there's no way to make it faster. As a side note, Ramshorn is even faster than Askama and does not need compiled templates: https://github.com/maciejhirsz/ramhorns https://github.com/maciejhirsz/ramhorns
  - Klonoar 6y ago
    Regarding your first sentence... I think we have the same conclusion. :) Thanks for the heads up re: Ramhorns. I'd seen it before but Mustache templates aren't particularly my thing - Django/Python guy in a previous life, so I've got love for the templating models from there. Willing to sacrifice a small amount of speed for developer ergonomics, things almost universally get rewritten once big enough scale comes into play anyway.
- chrismorgan 6y ago
  I don’t think recompiling for template changes is as unreasonable as you depict. It’s a standard technique in various languages, though I acknowledge that they’re mostly not languages that compile to native code, but rather an IL (e.g. ASP.NET with Razor) where the compilation time is somewhat lower. For a language like Rust, compilation times and tooling support mean that the technique is perhaps not ready for prime time, but it’s fundamentally a perfectly sound technique, and can produce wondrous developer ergonomics (again I cite Razor). There’s something quite incongruous about using dynamically-typed and basically uncheckable template language when you’re doing the rest of the code in Rust.
  - Klonoar 6y ago
    I didn't say that recompiling template changes is bad - e.g, you can accomplish the same result with much faster turnaround time using Tera, and for most projects the speed is in no way a bottleneck.
- littlestymaar 6y ago
  While I agree with your criticism of “cheating” and unnatural tricks that you are not likely to use in production (but it's more a fundamental criticism about this kind of benchmark games than a specific criticism of Actix), I don't really share you view on Askama: unless your templates change a lot (i.e. during initial development) or if you need to be able to change then dynamically in production without downtime, compiled templates like Askama are IMHO the way to go when going to production. You can and should use a interpreted one during initial development, and then switch when you're ready to go live. An actix-web + Askama user.
  - Klonoar 6y ago
    Using one during development and one during production is indeed a valid use case, but most people simply don't get anywhere close to that level of need. Let me be very clear: I like Askama and have submitted pull requests to the project in the past. For a new project where I need templates, I value the speed of reflected changes over compiling more stuff into the binary. There are also certain classes of problems with Askama that you can avoid with something like Tera - e.g, with Askama, if your global layout relies on a value passed in to the rendering context, you'll need to shove that value on every Template struct. Furthermore, most people will hit bottlenecks in other areas of their application long before templating becomes the issue (i.e, this was the same in Django-land too - templating via cached compiled Django or Jinja2 templates is almost never an issue). Askama is overkill until you need it. My recommendation, having built projects with both, is thus: build with Tera initially, and swap out hot paths to Askama down the road as they both use the same syntax (roughly) so it's not too difficult to convert one to the other. In a perfect world the projects would be merged, and you can use the Tera aspect in dev and get Askama in production for free... but that's easier said than done.
- brobdingnagians 6y ago
  Reminds me of see-phit[1] for C++, where it compiles the HTML to a tree at compile time. I guess that would be a similar way to get even more gains for dynamically created HTML. [1] https://github.com/rep-movsd/see-phit https://github.com/rep-movsd/see-phit
chiefsucker 6y ago
> "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest" I’m not sure I understand this, i. e. TRANTOR (which is the underlying network library for Drogon) recently merged in a patch that optimizes memory handling in a low level component of its event loop where object construction was minimized (first tests show another performance increase of about 5–10% on top of the current benchmark results). I fail to understand how such improvements have something to do with “all the latest methods and libraries”?

ori_b 6y ago

People ask for more features over time. I'd expect older frameworks to be more featureful, but slower.

Klonoar 6y ago

It's worth noting that actix-web is at the top by doing some insane tricks, which you wouldn't normally do in writing a web application. e.g at one point they used Askama as their templating engine (may have switched), which isn't really fair as using Askama means you force a recompile of your project for each template change - and I can tell you from experience that your compile times will shoot through the roof as well. Nobody is going to bother doing that. I say all of this as an actix-web user. It's my go-to. I just view these benchmarks as "if I ever needed to, I know I can wrangle performance out of it... but I won't assume it by default".

terhechte 6y ago
But most other projects on there do that as well. I regard tech empower as the maximum performance I can get with a given framework if need be. With actin I can add Askama, with Vapor, there's no way to make it faster. As a side note, Ramshorn is even faster than Askama and does not need compiled templates: https://github.com/maciejhirsz/ramhorns https://github.com/maciejhirsz/ramhorns
- Klonoar 6y ago
  Regarding your first sentence... I think we have the same conclusion. :) Thanks for the heads up re: Ramhorns. I'd seen it before but Mustache templates aren't particularly my thing - Django/Python guy in a previous life, so I've got love for the templating models from there. Willing to sacrifice a small amount of speed for developer ergonomics, things almost universally get rewritten once big enough scale comes into play anyway.
chrismorgan 6y ago
I don’t think recompiling for template changes is as unreasonable as you depict. It’s a standard technique in various languages, though I acknowledge that they’re mostly not languages that compile to native code, but rather an IL (e.g. ASP.NET with Razor) where the compilation time is somewhat lower. For a language like Rust, compilation times and tooling support mean that the technique is perhaps not ready for prime time, but it’s fundamentally a perfectly sound technique, and can produce wondrous developer ergonomics (again I cite Razor). There’s something quite incongruous about using dynamically-typed and basically uncheckable template language when you’re doing the rest of the code in Rust.
- Klonoar 6y ago
  I didn't say that recompiling template changes is bad - e.g, you can accomplish the same result with much faster turnaround time using Tera, and for most projects the speed is in no way a bottleneck.
littlestymaar 6y ago
While I agree with your criticism of “cheating” and unnatural tricks that you are not likely to use in production (but it's more a fundamental criticism about this kind of benchmark games than a specific criticism of Actix), I don't really share you view on Askama: unless your templates change a lot (i.e. during initial development) or if you need to be able to change then dynamically in production without downtime, compiled templates like Askama are IMHO the way to go when going to production. You can and should use a interpreted one during initial development, and then switch when you're ready to go live. An actix-web + Askama user.
- Klonoar 6y ago
  Using one during development and one during production is indeed a valid use case, but most people simply don't get anywhere close to that level of need. Let me be very clear: I like Askama and have submitted pull requests to the project in the past. For a new project where I need templates, I value the speed of reflected changes over compiling more stuff into the binary. There are also certain classes of problems with Askama that you can avoid with something like Tera - e.g, with Askama, if your global layout relies on a value passed in to the rendering context, you'll need to shove that value on every Template struct. Furthermore, most people will hit bottlenecks in other areas of their application long before templating becomes the issue (i.e, this was the same in Django-land too - templating via cached compiled Django or Jinja2 templates is almost never an issue). Askama is overkill until you need it. My recommendation, having built projects with both, is thus: build with Tera initially, and swap out hot paths to Askama down the road as they both use the same syntax (roughly) so it's not too difficult to convert one to the other. In a perfect world the projects would be merged, and you can use the Tera aspect in dev and get Askama in production for free... but that's easier said than done.
brobdingnagians 6y ago
Reminds me of see-phit[1] for C++, where it compiles the HTML to a tree at compile time. I guess that would be a similar way to get even more gains for dynamically created HTML. [1] https://github.com/rep-movsd/see-phit https://github.com/rep-movsd/see-phit

chiefsucker 6y ago

> "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest" I’m not sure I understand this, i. e. TRANTOR (which is the underlying network library for Drogon) recently merged in a patch that optimizes memory handling in a low level component of its event loop where object construction was minimized (first tests show another performance increase of about 5–10% on top of the current benchmark results). I fail to understand how such improvements have something to do with “all the latest methods and libraries”?