5 ms·

This is a pretty realistic balanced real-world benchmark of web frameworks (and languages): https://www.techempower.com/benchmarks/ https://www.techempower.com/

This is a pretty realistic balanced real-world benchmark of web frameworks (and languages): https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ It shows that Python with Django is literally 40 times slower than the fastest framework. Python with uvicorn is 10 times slower. The use of languages like Python and Ruby literally results in >10x the servers being used; which not only results in higher cost, but also greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels. Not to mention, dynamically-typed languages are truly horrible from a code readability point of view. Large code bases are IMO difficult to read and make sense of, hard to debug, and more prone to bugs, without static types. I'm aware that Elixir is dynamically-typed, but it (along with JS) is an exception in terms of speed. Most dynamically-typed languages are quite slow. Not only do dynamically-typed languages damage the environment as they're typically an order of magnitude slower, they also lower developer productivity, and damage the robustness and reliability of the software written in it. To be clear, I'm in favor of anything that increases productivity. If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it. I'm not sure how Elixir mitigates the downsides of dynamic typing (maybe lots of 'type checks' with pattern matching?), but it would definitely be super-nice if a well-designed (Kotlin or Haskell like?) statically-typed language targeting the BEAM existed...

dorfsmay 3y ago
Do you know if there are plans to publish more recent numbers, the newest one is a year old. I'll definitely take a look at JustJS, that's an impressive ranking!!!
- winter_blue 3y ago
  The creator of JustJS has written about how it's so fast actually: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01... (Related GitHub thread: https://github.com/just-js/just/issues/5 https://github.com/just-js/just/issues/5)
amarshall 3y ago
Sure, but the fastest Elixir-based entry on that list has half the score of uvicorn (Python), so not really sure what your point is here w.r.t. OP.
- brd529 3y ago
  Reading higher in the thread, some Elixir folks are saying that the techempower benchmarks used the wrong settings (debug mode, etc) for their Elixir benchmark.
  - amarshall 3y ago
    Though that doesn’t change the crux of my question (rather it reinforces it): why invoke that benchmark in this context at all?
    - winter_blue 3y ago
      The fastest Elixir framework on the list, phoenix, is about as fast as uvicorn. (Both around 10 times slower than dragon.) I was mostly responding directly to these 2 statements from my parent comment: > They did savings by re-implementing their services and attribute those savings to the new tool / programming language. > I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python. Which I didn't entirely agree with.
duckmysick 3y ago
Going from > greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels. to > dynamically-typed languages damage the environment as they're typically an order of magnitude slower is quite a stretch. Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels? Not that the appeal to the environment matters, because later on we have this: > If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it.
- winter_blue 3y ago
  > Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels? My opinion is that slow languages that use 10x the electricity, with no ROI for the 10x energy use is bad. High energy use, even if it's clean energy, implies a higher environmental toll. If a country were solely using nuclear and solar, higher energy use results in (1) more nuclear reactors constructed, and (2) more solar panels built. The manufacture and construction of both has an environment cost. Of course, with fossil fuels, the damage to the environment is potentially a lot worse. > Not that the appeal to the environment matters I don't think higher energy use is inherently bad. If we can improve the quality of life for human beings, then IMO a higher energy use is justified. I don't really believe in degrading our quality of life to lower our energy use. My problem with many popular slow languages is that they have a negative ROI for the higher electricity cost. In exchange for 10x the energy use, you have a language that results in less-readable code (a serious issue), that causes more bugs / less-reliable software, etc. We literally a get negative ROI in exchange for 10x the energy use. Which is absurd and illogical. If Hindley–Milner type inference had been more prevalent in the 1990s, I have a feeling dynamically-typed languages would have never take off. We're moving back to static typing with mypy, TypeScript, etc., but I'm hoping we move away entirely soon from using slow languages for writing servers serving large numbers of users. If we actually got something in exchange for it, it wouldn't bother me so much.
  - DoesntMatter22 3y ago
    For the vast majority of projects this makes no difference. If you are at Facebook scale? Sure. But then, you do what they did, write a VM to speed things up. Dynamic typing doesn't cost much more money on average and thankfully the cost of energy itself is a motivation for companies to do rewrites. If they are paying a lot in server costs and electricity then they do typically rewrite to reduce the amount of servers. For companies they primarily need to worry about running at a profit and getting to market quickly which dynamic languages do extremely well, and the costs in electricity and carbon aren't very high when your scale is small.
atraac 3y ago
> This is a pretty realistic balanced real-world benchmark of web frameworks (and languages): https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Did you even ever see implementation behind techempower benchmarks? There's NOTHING realistic in them. Those applications literally hardcode static content length header values to be faster. They are pretty good show of how low you can get to squeeze out performance but not one sane person will write code like that.
- winter_blue 3y ago
  I've only skimmed some of the code, but fwiw, anyone can read their code for each framework that's benchmarked here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... There's also a good amount of active discussion going on here: https://github.com/TechEmpower/FrameworkBenchmarks/discussions https://github.com/TechEmpower/FrameworkBenchmarks/discussio... Here's the code for the JS (just-js) benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/JavaScript/just/techempower.js https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... And here's the code for the Python uvicorn benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/uvicorn/app.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... From some light skimming, the JS and Python code doesn't look particularly abnormal or overly-optimized.
dlisboa 3y ago
Since you mentioned them I think it's worth telling people to take those benchmarks with a larger grain of salt. They've become such a pissing contest that I don't know if they can be called "real-world". There doesn't seem to be much scrutiny of the implementations. Take some Rust frameworks: they write out pre-computed header strings to the output buffer, completely bypassing what the framework's documentation recommends. Examples are actix[1] and ntex[2]. No one would ever do this in real life. Now I like Rust, and it'd likely be some of the fastest even without these shenanigans (Axum and may-http don't do that, I believe). But I don't know if other languages/frameworks have benchmarks implemented in the same non-idiomatic way just to look better. [1] - https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/actix/src/main_server.rs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2] -https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/ntex/src/main_plt.rs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
- winter_blue 3y ago
  Hmm, the pre-computed header strings is definitely interesting optimization, and does seem to be a bit non-idiomatic – but IMO, this is the sort of optimization the framework itself should try and do. If first several bytes of the header are going to identical for many requests, a super-optimized framework would ideally memoize it, and write it directly out, as this benchmark is doing. (And if the framework itself is doing it, then any user of the benchmark would just inherit that optimization, and not have to resort to non-idiomatic optimizations...)
Mawr 3y ago
> This is a pretty realistic balanced real-world benchmark of web frameworks (and languages): https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ I can't take seriously a benchmark that doesn't even show memory usage. Hahaha, yep: https://github.com/TechEmpower/FrameworkBenchmarks/issues/3966#issuecomment-511613377 https://github.com/TechEmpower/FrameworkBenchmarks/issues/39.... Nothing to see here.
- winter_blue 3y ago
  Ah, that's a good catch. Hmm, I wish Techempower listened to the issues brought up, and made the changes asked. There's unfortunately no other comparable as-comprehensive web framework benchmarks that I'm aware of.

dorfsmay 3y ago

Do you know if there are plans to publish more recent numbers, the newest one is a year old. I'll definitely take a look at JustJS, that's an impressive ranking!!!

winter_blue 3y ago
The creator of JustJS has written about how it's so fast actually: https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01... (Related GitHub thread: https://github.com/just-js/just/issues/5 https://github.com/just-js/just/issues/5)

amarshall 3y ago

Sure, but the fastest Elixir-based entry on that list has half the score of uvicorn (Python), so not really sure what your point is here w.r.t. OP.

brd529 3y ago
Reading higher in the thread, some Elixir folks are saying that the techempower benchmarks used the wrong settings (debug mode, etc) for their Elixir benchmark.
- amarshall 3y ago
  Though that doesn’t change the crux of my question (rather it reinforces it): why invoke that benchmark in this context at all?
  - winter_blue 3y ago
    The fastest Elixir framework on the list, phoenix, is about as fast as uvicorn. (Both around 10 times slower than dragon.) I was mostly responding directly to these 2 statements from my parent comment: > They did savings by re-implementing their services and attribute those savings to the new tool / programming language. > I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python. Which I didn't entirely agree with.

duckmysick 3y ago

Going from > greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels. to > dynamically-typed languages damage the environment as they're typically an order of magnitude slower is quite a stretch. Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels? Not that the appeal to the environment matters, because later on we have this: > If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it.

winter_blue 3y ago
> Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels? My opinion is that slow languages that use 10x the electricity, with no ROI for the 10x energy use is bad. High energy use, even if it's clean energy, implies a higher environmental toll. If a country were solely using nuclear and solar, higher energy use results in (1) more nuclear reactors constructed, and (2) more solar panels built. The manufacture and construction of both has an environment cost. Of course, with fossil fuels, the damage to the environment is potentially a lot worse. > Not that the appeal to the environment matters I don't think higher energy use is inherently bad. If we can improve the quality of life for human beings, then IMO a higher energy use is justified. I don't really believe in degrading our quality of life to lower our energy use. My problem with many popular slow languages is that they have a negative ROI for the higher electricity cost. In exchange for 10x the energy use, you have a language that results in less-readable code (a serious issue), that causes more bugs / less-reliable software, etc. We literally a get negative ROI in exchange for 10x the energy use. Which is absurd and illogical. If Hindley–Milner type inference had been more prevalent in the 1990s, I have a feeling dynamically-typed languages would have never take off. We're moving back to static typing with mypy, TypeScript, etc., but I'm hoping we move away entirely soon from using slow languages for writing servers serving large numbers of users. If we actually got something in exchange for it, it wouldn't bother me so much.
- DoesntMatter22 3y ago
  For the vast majority of projects this makes no difference. If you are at Facebook scale? Sure. But then, you do what they did, write a VM to speed things up. Dynamic typing doesn't cost much more money on average and thankfully the cost of energy itself is a motivation for companies to do rewrites. If they are paying a lot in server costs and electricity then they do typically rewrite to reduce the amount of servers. For companies they primarily need to worry about running at a profit and getting to market quickly which dynamic languages do extremely well, and the costs in electricity and carbon aren't very high when your scale is small.

atraac 3y ago

> This is a pretty realistic balanced real-world benchmark of web frameworks (and languages): https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Did you even ever see implementation behind techempower benchmarks? There's NOTHING realistic in them. Those applications literally hardcode static content length header values to be faster. They are pretty good show of how low you can get to squeeze out performance but not one sane person will write code like that.

winter_blue 3y ago
I've only skimmed some of the code, but fwiw, anyone can read their code for each framework that's benchmarked here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... There's also a good amount of active discussion going on here: https://github.com/TechEmpower/FrameworkBenchmarks/discussions https://github.com/TechEmpower/FrameworkBenchmarks/discussio... Here's the code for the JS (just-js) benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/JavaScript/just/techempower.js https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... And here's the code for the Python uvicorn benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/uvicorn/app.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... From some light skimming, the JS and Python code doesn't look particularly abnormal or overly-optimized.

dlisboa 3y ago

Since you mentioned them I think it's worth telling people to take those benchmarks with a larger grain of salt. They've become such a pissing contest that I don't know if they can be called "real-world". There doesn't seem to be much scrutiny of the implementations. Take some Rust frameworks: they write out pre-computed header strings to the output buffer, completely bypassing what the framework's documentation recommends. Examples are actix[1] and ntex[2]. No one would ever do this in real life. Now I like Rust, and it'd likely be some of the fastest even without these shenanigans (Axum and may-http don't do that, I believe). But I don't know if other languages/frameworks have benchmarks implemented in the same non-idiomatic way just to look better. [1] - https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/actix/src/main_server.rs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2] -https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/ntex/src/main_plt.rs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

winter_blue 3y ago
Hmm, the pre-computed header strings is definitely interesting optimization, and does seem to be a bit non-idiomatic – but IMO, this is the sort of optimization the framework itself should try and do. If first several bytes of the header are going to identical for many requests, a super-optimized framework would ideally memoize it, and write it directly out, as this benchmark is doing. (And if the framework itself is doing it, then any user of the benchmark would just inherit that optimization, and not have to resort to non-idiomatic optimizations...)

Mawr 3y ago

> This is a pretty realistic balanced real-world benchmark of web frameworks (and languages): https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ I can't take seriously a benchmark that doesn't even show memory usage. Hahaha, yep: https://github.com/TechEmpower/FrameworkBenchmarks/issues/3966#issuecomment-511613377 https://github.com/TechEmpower/FrameworkBenchmarks/issues/39.... Nothing to see here.

winter_blue 3y ago
Ah, that's a good catch. Hmm, I wish Techempower listened to the issues brought up, and made the changes asked. There's unfortunately no other comparable as-comprehensive web framework benchmarks that I'm aware of.