5 ms·

It is a big red flag to me if a project claims "fast", but does not prominently show benchmarks. Previously I have seen projects that claim to be the fastest wh

It is a big red flag to me if a project claims "fast", but does not prominently show benchmarks. Previously I have seen projects that claim to be the fastest which are 10x slower than alternative popular projects. There are two major benefits I can see from making public benchmarks: 1. showing which other projects you are competing with and 2. having a benchmark suite you keep an eye on helps you actually make your project fast.

jacob019 4y ago
I would also like to see some benchmarks, especially compared to aiohttp since that is what it appears to replace. I too am skeptical, aiohttp isn't all that fast, but it's execution time is negligible in a typical application. I found aiohttp to be about half the speed of gevent.pywsgi, but it turns out that doesn't matter at all when the vast majority of time is spent executing business logic and serializing responses.
mtrpcic 4y ago
This is funny, because often times as soon as you add benchmarks, the masses will scream that the benchmarks aren't useful because they don't test real world conditions (global network latency, wide ranges of queries with effectively random patterns, etc.). Benchmarks feel a bit like a "damned if you do, damned if you don't" situation. That said, I love me a good benchmark.
- thethirdone 4y ago
  There is even a comment on this page saying requests per second isn't everything. So even if you don't benchmark they will still say that benchmarks aren't useful. Of course I agree that benchmarks can't tell the whole story, but they can demonstrate huge differences in potential performance which can give you a reason to create a realistic benchmark for your use case.
- jammycrisp 4y ago
  For my own "fast" projects[1] I've taken to providing benchmarks, but adding a big 'ol caveat at the top describing ways in which the benchmark may not reflect reality. Some people really want to see benchmarks! Others are turned off by projects that overreach on performance claims. I'm hopeful that adding context around benchmarks to tamper excitement strikes the right balance. [1]: https://jcristharif.com/msgspec/benchmarks.html https://jcristharif.com/msgspec/benchmarks.html
eyegor 4y ago
I found a performance comparison here [0] and these are the same numbers that they present in a talk I found on youtube [1]. The gotcha I see is they haven't mentioned what version of each framework they're running, python runtime, and I don't see the code used for load testing so who knows what it's actually showing. However, being native code is a huge free win for performance so I don't doubt these numbers for a microbenchmark. I suspect the reason these aren't "front and center" on the github readme is because the project is very early and many of their commits mention performance. Feature wise it seems reasonably mature but the release cycle is quite fast and they've made a few breaking changes. If I were them, I'd wait until I was near 1.0 before putting performance claims in front too. [0] https://sansyrox.github.io/robyn/#/comparison https://sansyrox.github.io/robyn/#/comparison [1] https://youtu.be/QWzp6nFQxPQ https://youtu.be/QWzp6nFQxPQ
yuzuquat 4y ago
I was pretty curious about the project too (am not the author) and did some digging. The benchmark is here: https://sansyrox.github.io/robyn/#/comparison https://sansyrox.github.io/robyn/#/comparison I took a brief look into the internals and turns out it wraps actix web (https://github.com/sansyrox/robyn/blob/main/src/server.rs https://github.com/sansyrox/robyn/blob/main/src/server.rs), one of the more dominant web servers on techempower. tldr: I dont have a good reason to doubt the performance but seems pretty interesting if you want to use actix web without having to go through rust
prollyjethi 4y ago
Hey, The project is still in early stage and hence I don't show many comparisons. Here is a benchmark on TechEmpower - https://www.techempower.com/benchmarks/#section=test&runid=43469a25-670e-4bda-a3c3-652839435784&test=plaintext&l=hra0hr-35n&w=18y6o-0-0-0 https://www.techempower.com/benchmarks/#section=test&runid=4... I also have a comparison on the docs website: https://sansyrox.github.io/robyn/#/comparison https://sansyrox.github.io/robyn/#/comparison And my own benchmark code is https://github.com/sansyrox/robyn-comparison-benchmarks https://github.com/sansyrox/robyn-comparison-benchmarks

jacob019 4y ago

I would also like to see some benchmarks, especially compared to aiohttp since that is what it appears to replace. I too am skeptical, aiohttp isn't all that fast, but it's execution time is negligible in a typical application. I found aiohttp to be about half the speed of gevent.pywsgi, but it turns out that doesn't matter at all when the vast majority of time is spent executing business logic and serializing responses.

mtrpcic 4y ago

This is funny, because often times as soon as you add benchmarks, the masses will scream that the benchmarks aren't useful because they don't test real world conditions (global network latency, wide ranges of queries with effectively random patterns, etc.). Benchmarks feel a bit like a "damned if you do, damned if you don't" situation. That said, I love me a good benchmark.

thethirdone 4y ago
There is even a comment on this page saying requests per second isn't everything. So even if you don't benchmark they will still say that benchmarks aren't useful. Of course I agree that benchmarks can't tell the whole story, but they can demonstrate huge differences in potential performance which can give you a reason to create a realistic benchmark for your use case.
jammycrisp 4y ago
For my own "fast" projects[1] I've taken to providing benchmarks, but adding a big 'ol caveat at the top describing ways in which the benchmark may not reflect reality. Some people really want to see benchmarks! Others are turned off by projects that overreach on performance claims. I'm hopeful that adding context around benchmarks to tamper excitement strikes the right balance. [1]: https://jcristharif.com/msgspec/benchmarks.html https://jcristharif.com/msgspec/benchmarks.html

eyegor 4y ago

I found a performance comparison here [0] and these are the same numbers that they present in a talk I found on youtube [1]. The gotcha I see is they haven't mentioned what version of each framework they're running, python runtime, and I don't see the code used for load testing so who knows what it's actually showing. However, being native code is a huge free win for performance so I don't doubt these numbers for a microbenchmark. I suspect the reason these aren't "front and center" on the github readme is because the project is very early and many of their commits mention performance. Feature wise it seems reasonably mature but the release cycle is quite fast and they've made a few breaking changes. If I were them, I'd wait until I was near 1.0 before putting performance claims in front too. [0] https://sansyrox.github.io/robyn/#/comparison https://sansyrox.github.io/robyn/#/comparison [1] https://youtu.be/QWzp6nFQxPQ https://youtu.be/QWzp6nFQxPQ

yuzuquat 4y ago

I was pretty curious about the project too (am not the author) and did some digging. The benchmark is here: https://sansyrox.github.io/robyn/#/comparison https://sansyrox.github.io/robyn/#/comparison I took a brief look into the internals and turns out it wraps actix web (https://github.com/sansyrox/robyn/blob/main/src/server.rs https://github.com/sansyrox/robyn/blob/main/src/server.rs), one of the more dominant web servers on techempower. tldr: I dont have a good reason to doubt the performance but seems pretty interesting if you want to use actix web without having to go through rust

prollyjethi 4y ago

Hey, The project is still in early stage and hence I don't show many comparisons. Here is a benchmark on TechEmpower - https://www.techempower.com/benchmarks/#section=test&runid=43469a25-670e-4bda-a3c3-652839435784&test=plaintext&l=hra0hr-35n&w=18y6o-0-0-0 https://www.techempower.com/benchmarks/#section=test&runid=4... I also have a comparison on the docs website: https://sansyrox.github.io/robyn/#/comparison https://sansyrox.github.io/robyn/#/comparison And my own benchmark code is https://github.com/sansyrox/robyn-comparison-benchmarks https://github.com/sansyrox/robyn-comparison-benchmarks