6 ms·

The author raises interesting points, but none of them are novel. The general tone here seems to be along the lines of: there are just so many moving variables

The author raises interesting points, but none of them are novel. The general tone here seems to be along the lines of: there are just so many moving variables, no fair comparisons may be fairly made. But I judge such positions agains this heuristic: "But, is it actionable?" Pragmatism demands we at least take a glance at languages, frameworks, and platforms from the perfromance perspective. Here is a practical example: There may be some issues with the Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. c++, js, amd rust dominate the top 20% and the differences in requests-per-second among them, statistically less significant, literally engulf the total rps numbers for the python or ruby rps totals. If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information.

Retric 5y ago
That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query.
- omegalulw 5y ago
  Not always. C++/C are also typically better in memory footprint. You can run more threads in the same server, so better qps.
- jzoch 5y ago
  Until the data store you grabbed is written in Ruby too :)
- PragmaticPulp 5y ago
  > That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query. Some of the TechEmpower benchmarks have a significant database component as well. Obviously if you have database queries taking 0.5s or more then it's a different story, but in my experience the server language/framework still has a large impact on total performance even with the database in the equation. In my experience, memory footprint is also a huge variable. For example, I rewrote a certain gRPC service in Rust and cut the memory usage by almost 80% relative to the initial implementation in a higher level language. We weren't bottlenecked on requests/second performance but being able to scale down our memory usage had a huge impact on cloud costs.
- yen223 5y ago
  There's the Fortunes benchmark, that involves reading off a database: https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r20&hw=... Something to observe from those benchmarks is that "fast" languages are still fast in those benchmarks. The server isn't waiting on I/O 100% of the time - it still has to do serialization and parsing and all other CPU-bound tasks, and those still benefit from a "faster" language.
- tylerhou 5y ago
  You can hide latency with sufficient parallelism. Since read requests are trivially parallelizable, as long as your database queries are mostly reads the database latency is probably not the limiting factor for throughput. (Even if your C++ program processes too many requests for one replica to handle, it's not hard to round-robin to multiple database replicas.)
- ksec 5y ago
  Filtered in Fortunes Cookies, Full Stack, ORM Only, MySQL / Postgres DB, and Realistic Approach, the difference is still 50x. That is already lower than the RAW SQL approach from 100 to 150x. And if we just look at .Net and ASP, these were done before all the optimisation came in latest .Net and ASP.net version.
- ezekiel68 5y ago
  This is a good point, and there are ways to take advantage of this. The c++ frameworks that beat rust advertise in their repos that they are using a forked PostgreSQL DB driver which has been re-written to be asynchronous. There's a js framework named just-js which tops the current (nightly) DB Update benchmark on TechEmpower and I suspect they are doing the same. Additionally, Java, python, and ruby frameworks tend to gobble RAM (I work in a large corp with many deployed Java web services). This requires instances to autoscale based on mem usage or to use instances with more mem. Rust and c++ frameworks often utilize a fraction of the memory for the same service (yes, we've tested this). So we can use smaller and/or fewer tasks/instances in our autoscaling pools.
derbOac 5y ago
I tend to think of these things in terms of some limit approaching reality, whatever that is, as the benchmarks continue. If you have an open competition, and there's enough editing and revision by enough actors over time, it gives you some idea of how fast something is likely to be in practice. That open scrutiny and process of suggested revision by experts in a language is a rough sampling of what is likely to be encountered in practice, with a bunch of experts chipping away at the problem. If a difference in times exists, that's about as good as you're likely to get, with further time differences decreasing in size and probability as the whole process continues. I'd even probably argue that even if you have a bunch of naive implementations, it still tells you about what's likely to be encountered in practice, because those naive implementations are often not so naive, and many programmers are more naive than you would like to be the case. Finally, I think in a lot of cases, it's clearer how to do a certain benchmark in a certain language than the essay assumes. Sure, a complex one probably might require some unusual level of expertise. But many benchmarks are fairly small and circumscribed, for better or worse, and consequently pretty tractable.
lmm 5y ago
> If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away You're almost certainly not. You almost certainly are nowhere near the RPS limit. You almost certainly are bottlenecked on your own code far more than your framework/language. You're almost certainly running at a performance level of "just barely not bad enough to spend time improving" and would almost certainly be doing the same in any other language. You're probably using a slower framework than the fastest python or ruby framework, so the fact that faster frameworks exist in your language is irrelevant. And if you are in the (rare) case where you're handling enough requests for this choice to matter, then you're big enough to benchmark some proper prototypes that measure the kind of thing your application actually needs to do. Almost anything is a better use of your time than looking at the benchmarks. And if you are going to look at the benchmarks, you shouldn't look at the RPS total; what you should do is first estimate how many requests your app needs to handle and then apply that as a yes/no filter to the frameworks you're considering. "It can handle 1000x as many requests as we need instead of 100x" is not a good reason to pick a framework, much less a language.
- ezekiel68 5y ago
  I wish this were true -- but it isn't. Partly because Java and scripting frameworks can easily get memory bound instead of CPU bound. I've worked for several large enterprises running externally accessible services in the cloud (and this is true of my current employer). Of course, all of our services have at least 2 "tasks" or instances to help with fault tolerance, yet most of them do, in fact, auto-scale up to more instances during a day based on either RAM or CPU load. (for the curious, apps which read from message queues tend to get CPU bound while web services in Java or a scripting language tend to get memory bound) If you must run 3 instances instead of 2, that is 50% more instance costs for that time period, etc. Moving a service from Java 11 to rust brought the memory load down sufficiently that we didn't need to autoscale at all. Some other service might autoscale, but to fewer max instances. And this adds up multiplied by days by services.
  - lmm 5y ago
    It takes a pretty extreme level of fine tuning to hit that window - Java memory usage is generally about 2x equivalent Rust. (It's completely possible your rewrite saved much more than that - but in that case, doing the same rewrite in Java would probably have brought you similar gains). And unless your user number is very static, you're not going to be in that window for long - maybe you push back the point at which you have to implement autoscaling / do some serious profiling work / start shopping around cloud vendors / ... by six months, or a year, which is helpful but not a huge return on rewriting your whole system. A factor of 10x or 100x is maybe worth letting it determine your choice of language. But given that most performance differences come from algorithm selection rather than the low-level details of your runtime, a language that makes it easy to understand what your code is doing is definitely worth giving up a factor of 2x or even 5x for. For a mature codebase where your userbase is not really growing any more, that kind of marginal performance work can be worthwhile - maybe rather than putting off the point at which you have to scale out, it lets you avoid having to scale out at all and saves you 50% on your running costs forever. But that kind of environment is the same kind of environment where it makes sense to put the codebase into maintenance mode and get rid of most of the expensive programmers (perhaps replacing them with outsourcing).
slezyr 5y ago
> Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. Have you seen the source code for the Django app there and one of the top performers? The Django app is just a simple example any developer would write. While Rust's top performers are full of hacks like checking the N-th letter of the HTTP method, no other person would use it in daily life.
- maverwa 5y ago
  While what you say is true, I don’t see how this invalidates the thesis of the parent comment. Two examples do not invalidate the general trend there. If, however, all implementations are flawed like that, we are having a different problem, right? But yes, super-optimized cases should be removed or marked as such.
  - slezyr 5y ago
    > If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. No one "throws away money" based on the Tech Empower benchmark. Because many of those optimizations are not free and, often, are not available. Such micro-optimizations are present in every program. This benchmark is just noise at this point and should be used as an argument against any framework/language. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/plugins/SyncPlugin.cc#L21 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
- ezekiel68 5y ago
  This is a common claim -- but no one has yet provided any evidence to back it up. So I got curious and took a look in the TechEmpower/FrameworkBenchmarks repo up on github. For the top rust frameworks (actix and ntex), the submissions are surprisingly short and easily comprehendable. Those two, at least, don't pull anything particularly tricky other than perhaps caching the initial portion of a successful HTTP response string (they all do that). By the same token, there has not been any evidence I have seen to support the claim that the scripting language frameworks have not also done their best with their submissions.
ashtonkem 5y ago
> If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. It’s worth pointing out that in some cases developer productivity can swamp server cost. It might be reasonable to pay the server bill necessary to develop in Ruby rather than C++, for example. Obviously the relationship between performance and developer productivity isn’t strictly linear.
- pclmulqdq 5y ago
  I recently wanted to build a serverless application, but as far as I can tell, no serverless platform takes code in C or C++, so I was stuck working in a language I didn't know well. There may be an opportunity for someone to build a serverless platform that uses verified functions in C/C++/Rust, so you can get both performance and convenience.
  - ezekiel68 5y ago
    This used to be harder -- you would set the lambda up for the go language runtime, but send your own statically linked binary with a special name instead of a compiled go artifact. But now AWS offers Custom Runtimes[0]. So you can set up the infra using some convenient tool such as AWS CloudFormation templates, terraform, or ansible, and then deploy your custom runtime c/c++ build from there (again, using ansible or AWS CLI or whatever). If you're using rust, there are now ways to make that work, including plugins for the popular "serverless" (sls) framework to deploy rust builds[1] [0] https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom... [1] https://github.com/awslabs/aws-lambda-rust-runtime https://github.com/awslabs/aws-lambda-rust-runtime
nonameiguess 5y ago
This raises an interesting question to me. When you look at high-performance Python libraries, they achieve the performance by just offloading the critical paths to compiled C modules. I remember reading some comparison of aiohttp to other Python frameworks a while back where the author expected aiohttp to be faster since it doesn't need to block on the network, but it wasn't, largely because of how slow it is to parse http requests in Python. If the bottleneck in serving a ton of requests per second is that Python is bad at serializing and deserializing http headers, why has no one just written a package that offloads that part of the library to a compiled C module? There is seemingly nothing preventing maintaining the convenience of Python for implementing your high-level business logic, but moving the bottlenecks to C. That has been exactly what programmers in scientific computing fields have been doing for 30 years and why projects like NumPy exist in the first place. So why do Python web programmers not just do the same thing? Heck, aiohttp already does this for the event loop, which just uses libuv the same as Node does. Both teams recognize that is a critical path component and it would be really slow to implement it in either Python or Javascript.

Retric 5y ago

That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query.

omegalulw 5y ago
Not always. C++/C are also typically better in memory footprint. You can run more threads in the same server, so better qps.
jzoch 5y ago
Until the data store you grabbed is written in Ruby too :)
PragmaticPulp 5y ago
> That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query. Some of the TechEmpower benchmarks have a significant database component as well. Obviously if you have database queries taking 0.5s or more then it's a different story, but in my experience the server language/framework still has a large impact on total performance even with the database in the equation. In my experience, memory footprint is also a huge variable. For example, I rewrote a certain gRPC service in Rust and cut the memory usage by almost 80% relative to the initial implementation in a higher level language. We weren't bottlenecked on requests/second performance but being able to scale down our memory usage had a huge impact on cloud costs.
yen223 5y ago
There's the Fortunes benchmark, that involves reading off a database: https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r20&hw=... Something to observe from those benchmarks is that "fast" languages are still fast in those benchmarks. The server isn't waiting on I/O 100% of the time - it still has to do serialization and parsing and all other CPU-bound tasks, and those still benefit from a "faster" language.
tylerhou 5y ago
You can hide latency with sufficient parallelism. Since read requests are trivially parallelizable, as long as your database queries are mostly reads the database latency is probably not the limiting factor for throughput. (Even if your C++ program processes too many requests for one replica to handle, it's not hard to round-robin to multiple database replicas.)
ksec 5y ago
Filtered in Fortunes Cookies, Full Stack, ORM Only, MySQL / Postgres DB, and Realistic Approach, the difference is still 50x. That is already lower than the RAW SQL approach from 100 to 150x. And if we just look at .Net and ASP, these were done before all the optimisation came in latest .Net and ASP.net version.
ezekiel68 5y ago
This is a good point, and there are ways to take advantage of this. The c++ frameworks that beat rust advertise in their repos that they are using a forked PostgreSQL DB driver which has been re-written to be asynchronous. There's a js framework named just-js which tops the current (nightly) DB Update benchmark on TechEmpower and I suspect they are doing the same. Additionally, Java, python, and ruby frameworks tend to gobble RAM (I work in a large corp with many deployed Java web services). This requires instances to autoscale based on mem usage or to use instances with more mem. Rust and c++ frameworks often utilize a fraction of the memory for the same service (yes, we've tested this). So we can use smaller and/or fewer tasks/instances in our autoscaling pools.

derbOac 5y ago

I tend to think of these things in terms of some limit approaching reality, whatever that is, as the benchmarks continue. If you have an open competition, and there's enough editing and revision by enough actors over time, it gives you some idea of how fast something is likely to be in practice. That open scrutiny and process of suggested revision by experts in a language is a rough sampling of what is likely to be encountered in practice, with a bunch of experts chipping away at the problem. If a difference in times exists, that's about as good as you're likely to get, with further time differences decreasing in size and probability as the whole process continues. I'd even probably argue that even if you have a bunch of naive implementations, it still tells you about what's likely to be encountered in practice, because those naive implementations are often not so naive, and many programmers are more naive than you would like to be the case. Finally, I think in a lot of cases, it's clearer how to do a certain benchmark in a certain language than the essay assumes. Sure, a complex one probably might require some unusual level of expertise. But many benchmarks are fairly small and circumscribed, for better or worse, and consequently pretty tractable.

lmm 5y ago

> If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away You're almost certainly not. You almost certainly are nowhere near the RPS limit. You almost certainly are bottlenecked on your own code far more than your framework/language. You're almost certainly running at a performance level of "just barely not bad enough to spend time improving" and would almost certainly be doing the same in any other language. You're probably using a slower framework than the fastest python or ruby framework, so the fact that faster frameworks exist in your language is irrelevant. And if you are in the (rare) case where you're handling enough requests for this choice to matter, then you're big enough to benchmark some proper prototypes that measure the kind of thing your application actually needs to do. Almost anything is a better use of your time than looking at the benchmarks. And if you are going to look at the benchmarks, you shouldn't look at the RPS total; what you should do is first estimate how many requests your app needs to handle and then apply that as a yes/no filter to the frameworks you're considering. "It can handle 1000x as many requests as we need instead of 100x" is not a good reason to pick a framework, much less a language.

ezekiel68 5y ago
I wish this were true -- but it isn't. Partly because Java and scripting frameworks can easily get memory bound instead of CPU bound. I've worked for several large enterprises running externally accessible services in the cloud (and this is true of my current employer). Of course, all of our services have at least 2 "tasks" or instances to help with fault tolerance, yet most of them do, in fact, auto-scale up to more instances during a day based on either RAM or CPU load. (for the curious, apps which read from message queues tend to get CPU bound while web services in Java or a scripting language tend to get memory bound) If you must run 3 instances instead of 2, that is 50% more instance costs for that time period, etc. Moving a service from Java 11 to rust brought the memory load down sufficiently that we didn't need to autoscale at all. Some other service might autoscale, but to fewer max instances. And this adds up multiplied by days by services.
- lmm 5y ago
  It takes a pretty extreme level of fine tuning to hit that window - Java memory usage is generally about 2x equivalent Rust. (It's completely possible your rewrite saved much more than that - but in that case, doing the same rewrite in Java would probably have brought you similar gains). And unless your user number is very static, you're not going to be in that window for long - maybe you push back the point at which you have to implement autoscaling / do some serious profiling work / start shopping around cloud vendors / ... by six months, or a year, which is helpful but not a huge return on rewriting your whole system. A factor of 10x or 100x is maybe worth letting it determine your choice of language. But given that most performance differences come from algorithm selection rather than the low-level details of your runtime, a language that makes it easy to understand what your code is doing is definitely worth giving up a factor of 2x or even 5x for. For a mature codebase where your userbase is not really growing any more, that kind of marginal performance work can be worthwhile - maybe rather than putting off the point at which you have to scale out, it lets you avoid having to scale out at all and saves you 50% on your running costs forever. But that kind of environment is the same kind of environment where it makes sense to put the codebase into maintenance mode and get rid of most of the expensive programmers (perhaps replacing them with outsourcing).

slezyr 5y ago

> Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. Have you seen the source code for the Django app there and one of the top performers? The Django app is just a simple example any developer would write. While Rust's top performers are full of hacks like checking the N-th letter of the HTTP method, no other person would use it in daily life.

maverwa 5y ago
While what you say is true, I don’t see how this invalidates the thesis of the parent comment. Two examples do not invalidate the general trend there. If, however, all implementations are flawed like that, we are having a different problem, right? But yes, super-optimized cases should be removed or marked as such.
- slezyr 5y ago
  > If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. No one "throws away money" based on the Tech Empower benchmark. Because many of those optimizations are not free and, often, are not available. Such micro-optimizations are present in every program. This benchmark is just noise at this point and should be used as an argument against any framework/language. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/plugins/SyncPlugin.cc#L21 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
ezekiel68 5y ago
This is a common claim -- but no one has yet provided any evidence to back it up. So I got curious and took a look in the TechEmpower/FrameworkBenchmarks repo up on github. For the top rust frameworks (actix and ntex), the submissions are surprisingly short and easily comprehendable. Those two, at least, don't pull anything particularly tricky other than perhaps caching the initial portion of a successful HTTP response string (they all do that). By the same token, there has not been any evidence I have seen to support the claim that the scripting language frameworks have not also done their best with their submissions.

ashtonkem 5y ago

> If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. It’s worth pointing out that in some cases developer productivity can swamp server cost. It might be reasonable to pay the server bill necessary to develop in Ruby rather than C++, for example. Obviously the relationship between performance and developer productivity isn’t strictly linear.

pclmulqdq 5y ago
I recently wanted to build a serverless application, but as far as I can tell, no serverless platform takes code in C or C++, so I was stuck working in a language I didn't know well. There may be an opportunity for someone to build a serverless platform that uses verified functions in C/C++/Rust, so you can get both performance and convenience.
- ezekiel68 5y ago
  This used to be harder -- you would set the lambda up for the go language runtime, but send your own statically linked binary with a special name instead of a compiled go artifact. But now AWS offers Custom Runtimes[0]. So you can set up the infra using some convenient tool such as AWS CloudFormation templates, terraform, or ansible, and then deploy your custom runtime c/c++ build from there (again, using ansible or AWS CLI or whatever). If you're using rust, there are now ways to make that work, including plugins for the popular "serverless" (sls) framework to deploy rust builds[1] [0] https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom... [1] https://github.com/awslabs/aws-lambda-rust-runtime https://github.com/awslabs/aws-lambda-rust-runtime

nonameiguess 5y ago

This raises an interesting question to me. When you look at high-performance Python libraries, they achieve the performance by just offloading the critical paths to compiled C modules. I remember reading some comparison of aiohttp to other Python frameworks a while back where the author expected aiohttp to be faster since it doesn't need to block on the network, but it wasn't, largely because of how slow it is to parse http requests in Python. If the bottleneck in serving a ton of requests per second is that Python is bad at serializing and deserializing http headers, why has no one just written a package that offloads that part of the library to a compiled C module? There is seemingly nothing preventing maintaining the convenience of Python for implementing your high-level business logic, but moving the bottlenecks to C. That has been exactly what programmers in scientific computing fields have been doing for 30 years and why projects like NumPy exist in the first place. So why do Python web programmers not just do the same thing? Heck, aiohttp already does this for the event loop, which just uses libuv the same as Node does. Both teams recognize that is a critical path component and it would be really slow to implement it in either Python or Javascript.