6 ms·

Lies, damn lies and benchmarks. The first benchmark is pretty much an empty request, so it measure each framework "overhead". That shows Rails+Puma at `0.25ms`

by byroot 3y ago

Lies, damn lies and benchmarks. The first benchmark is pretty much an empty request, so it measure each framework "overhead". That shows Rails+Puma at `0.25ms` overhead per request, and rage+iodine at `0.013ms`. That is significantly less yes, but it the overwhelming majority of case it absolutely don't matter, and this difference mostly come from Rails providing extra features. As for the second one, it's entirely irrelevant. It's benchmarking a purely IO workload with Puma caped at 5 threads vs Iodine without a concurrency cap. You can increase "Rails" throughput about as much as you want in that benchmark by raising the number of Puma threads or switching to Falcon. But also, not capping concurrency like this is risky. The second some CPU heavy work is mixed with this pure IO work, latency will suffer terribly. It's fine for a micro-service where you know it's really all IO though. Disclaimer: I'm a member of Rails, but I welcome alternative frameworks, I'm just very annoyed at how terrible most benchmarks are.

alberth 3y ago
Slightly off topic: what is the preferred way to deploy Rails these days? Is it to use Puma? Unicorn, etc?
- PikachuEXE 3y ago
  Puma works fine most of the time (config it accordingly of course No idea what other options are out there though
  - gvurrdon 3y ago
    I've been using Nginx/Passenger: https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/ https://www.phusionpassenger.com/library/deploy/nginx/deploy...
    - PikachuEXE 3y ago
      I actually switch from Passenger (to puma But then I got reverse proxy (caddy) in front
- xiljin 3y ago
  There are several options but Puma seems to be the most popular at the moment. A derivative of Unicorn named Pitchfork is also in the works from Shopify - https://github.com/Shopify/pitchfork https://github.com/Shopify/pitchfork
- compumike 3y ago
  We use Puma in production.
boundlessdreamz 3y ago
Also this doesn't show how database access is handled which is the hard part. If you are not touching the database, you can run Rails on falcon and get fiber based concurrency. If you run falcon on rails and access database, then you have to explicitly checkin/checkout a connection to be safe. Details here - https://github.com/rails/rails/issues/42271 https://github.com/rails/rails/issues/42271.
ricardobeat 3y ago
> It's benchmarking a purely IO workload with Puma caped at 5 threads vs Iodine without a concurrency cap It appears the benchmark sets the Iodine thread count to 1 as the legend says. Do you think that’s not the case? Facil.io uses an event loop to handle high concurrency in a single thread, results are not out if line with code that used LibUV.
- byroot 3y ago
  That is precisely my point. One has concurrency capped at 5, the other has no cap (because it uses an event loop). The Puma process is likely 99% idle in that benchmark, you could crank up the thread count to 50 and get (not quite) 10x better results. An event loop is definitely more efficient than a thread pool for such a work-load, I'm not denying that, but in this specific instance Rails+Puma is configured (granted it's just the defaults) with ball and chain. I'm not saying the benchmark is dishonest, I'm saying it makes no sense.
  - ricardobeat 3y ago
    That objection makes no sense. The fact that Puma is idle most of the time reflects a disadvantage. The event loop makes much better usage of a single thread, and that shows in the benchmark. It's like saying comparing Node to Rhino, or Vertx vs Spring Boot is unfair. Better performance per thread / core is the whole point. You will not achieve the same level of performance with threads except in low-volume benchmarks, and _especially_ in any kind of cloud environment where you usually only have a single core and a shared kernel.
    - byroot 3y ago
      My point is that event loop are not inherently superior to thread pools. When you have a workload that is CPU heavy enough that you won't have more than a handful of threads of concurrency, using a thread pool will allow preemption, hence give you a better tail latency, etc. It's all tradeoff, event loops are good for some work loads, threads for others. So to make an analogy of my own, it's like making a race between an F1 car and a semi-truck. They are both useful and efficient motor vehicule, but they're not optimized for the same thing.
andai 3y ago
So what would be a better benchmark? Perhaps a "standard" "real world" app, like https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld > The RealWorld Demo is a large community-backed open source project that showcases all web frameworks client or server-side through a building an example app that is much more substantial than the typical TodoMVC toy demos or the stripped out scenarios you find in benchmarks. As Conduit, a Medium clone of sorts, your application has to handle real issues like authentication, routing, and async data loading. It has a standardized specification [...] Or something simpler?
- byroot 3y ago
  > So what would be a better benchmark? It all depends on what you are trying to demonstrate. The current benchmarks would be fine if properly contextualized and presented a bit differently. e.g. for the first one, instead of representing it in request/second, do a breakdown of average/p50/p90/p99 latency over X requests. That's both much more interesting data, and much less misleading. Even the second one could be fine if used to demonstrate why an event loop can be the appropriate thing to use for IO heavy workloads. But here they are just included with no explanations, implying a blanket "we're 25 times faster than Rails".
- whalesalad 3y ago
  TechEmpower has a few different classes of benchmark. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Off the top of my head: - json serialization - fetching random objects from an actual mysql/psql database - cached queries - performing mutations / data updates writing "hello world" as a response is naturally going to do 75k per second
chitinbottler0a 3y ago
I totally see how you might find these benchmarks irrelevant or "unfair". But there are no lies - I didn't make those numbers up. These are pretty standard benchmarks showing the overhead of a framework and how it behaves with certain types of operations.
PH95VuimJjqBqy 3y ago
My whole thing is that ruby and performance don't mix, choose something else. I'm not saying don't bother optimizing something written in Ruby, I'm saying Ruby has inherent limitations in terms of performance. If performance is your goal you're using the wrong language.
neonsunset 3y ago
There is certainly a lot of speculation in Techempower benchmarks and top entries can utilize questionable techniques like simply writing a byte array literal to output stream instead of constructing a response, or (in the past) DB query coalescing to work around inherent limitations of the DB in case of Fortunes or DB quries. And yet, the fastest Ruby entry is at 274th place while Rails is at 427th. https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...
rajaravivarma_r 3y ago
Except, increasing the threads increases the memory usage proportionally. From what I have seen, the memory won't be released back and it keeps increasing until the process gets OOM'ed. At least this was the case with Ruby 3.0 and 3.2.

by byroot 3y ago

alberth 3y ago

Slightly off topic: what is the preferred way to deploy Rails these days? Is it to use Puma? Unicorn, etc?

PikachuEXE 3y ago
Puma works fine most of the time (config it accordingly of course No idea what other options are out there though
- gvurrdon 3y ago
  I've been using Nginx/Passenger: https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/ https://www.phusionpassenger.com/library/deploy/nginx/deploy...
  - PikachuEXE 3y ago
    I actually switch from Passenger (to puma But then I got reverse proxy (caddy) in front
xiljin 3y ago
There are several options but Puma seems to be the most popular at the moment. A derivative of Unicorn named Pitchfork is also in the works from Shopify - https://github.com/Shopify/pitchfork https://github.com/Shopify/pitchfork
compumike 3y ago
We use Puma in production.

boundlessdreamz 3y ago

Also this doesn't show how database access is handled which is the hard part. If you are not touching the database, you can run Rails on falcon and get fiber based concurrency. If you run falcon on rails and access database, then you have to explicitly checkin/checkout a connection to be safe. Details here - https://github.com/rails/rails/issues/42271 https://github.com/rails/rails/issues/42271.

ricardobeat 3y ago

> It's benchmarking a purely IO workload with Puma caped at 5 threads vs Iodine without a concurrency cap It appears the benchmark sets the Iodine thread count to 1 as the legend says. Do you think that’s not the case? Facil.io uses an event loop to handle high concurrency in a single thread, results are not out if line with code that used LibUV.

byroot 3y ago
That is precisely my point. One has concurrency capped at 5, the other has no cap (because it uses an event loop). The Puma process is likely 99% idle in that benchmark, you could crank up the thread count to 50 and get (not quite) 10x better results. An event loop is definitely more efficient than a thread pool for such a work-load, I'm not denying that, but in this specific instance Rails+Puma is configured (granted it's just the defaults) with ball and chain. I'm not saying the benchmark is dishonest, I'm saying it makes no sense.
- ricardobeat 3y ago
  That objection makes no sense. The fact that Puma is idle most of the time reflects a disadvantage. The event loop makes much better usage of a single thread, and that shows in the benchmark. It's like saying comparing Node to Rhino, or Vertx vs Spring Boot is unfair. Better performance per thread / core is the whole point. You will not achieve the same level of performance with threads except in low-volume benchmarks, and _especially_ in any kind of cloud environment where you usually only have a single core and a shared kernel.
  - byroot 3y ago
    My point is that event loop are not inherently superior to thread pools. When you have a workload that is CPU heavy enough that you won't have more than a handful of threads of concurrency, using a thread pool will allow preemption, hence give you a better tail latency, etc. It's all tradeoff, event loops are good for some work loads, threads for others. So to make an analogy of my own, it's like making a race between an F1 car and a semi-truck. They are both useful and efficient motor vehicule, but they're not optimized for the same thing.

andai 3y ago

So what would be a better benchmark? Perhaps a "standard" "real world" app, like https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld > The RealWorld Demo is a large community-backed open source project that showcases all web frameworks client or server-side through a building an example app that is much more substantial than the typical TodoMVC toy demos or the stripped out scenarios you find in benchmarks. As Conduit, a Medium clone of sorts, your application has to handle real issues like authentication, routing, and async data loading. It has a standardized specification [...] Or something simpler?

byroot 3y ago
> So what would be a better benchmark? It all depends on what you are trying to demonstrate. The current benchmarks would be fine if properly contextualized and presented a bit differently. e.g. for the first one, instead of representing it in request/second, do a breakdown of average/p50/p90/p99 latency over X requests. That's both much more interesting data, and much less misleading. Even the second one could be fine if used to demonstrate why an event loop can be the appropriate thing to use for IO heavy workloads. But here they are just included with no explanations, implying a blanket "we're 25 times faster than Rails".
whalesalad 3y ago
TechEmpower has a few different classes of benchmark. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Off the top of my head: - json serialization - fetching random objects from an actual mysql/psql database - cached queries - performing mutations / data updates writing "hello world" as a response is naturally going to do 75k per second

chitinbottler0a 3y ago

I totally see how you might find these benchmarks irrelevant or "unfair". But there are no lies - I didn't make those numbers up. These are pretty standard benchmarks showing the overhead of a framework and how it behaves with certain types of operations.

PH95VuimJjqBqy 3y ago

My whole thing is that ruby and performance don't mix, choose something else. I'm not saying don't bother optimizing something written in Ruby, I'm saying Ruby has inherent limitations in terms of performance. If performance is your goal you're using the wrong language.

neonsunset 3y ago

There is certainly a lot of speculation in Techempower benchmarks and top entries can utilize questionable techniques like simply writing a byte array literal to output stream instead of constructing a response, or (in the past) DB query coalescing to work around inherent limitations of the DB in case of Fortunes or DB quries. And yet, the fastest Ruby entry is at 274th place while Rails is at 427th. https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

rajaravivarma_r 3y ago

Except, increasing the threads increases the memory usage proportionally. From what I have seen, the memory won't be released back and it keeps increasing until the process gets OOM'ed. At least this was the case with Ruby 3.0 and 3.2.