6 ms·

Why most programming language performance comparisons are most likely wrong

42 points by Sindisil 5y ago

ezekiel68 5y ago
The author raises interesting points, but none of them are novel. The general tone here seems to be along the lines of: there are just so many moving variables, no fair comparisons may be fairly made. But I judge such positions agains this heuristic: "But, is it actionable?" Pragmatism demands we at least take a glance at languages, frameworks, and platforms from the perfromance perspective. Here is a practical example: There may be some issues with the Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. c++, js, amd rust dominate the top 20% and the differences in requests-per-second among them, statistically less significant, literally engulf the total rps numbers for the python or ruby rps totals. If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information.
- Retric 5y ago
  That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query.
  - omegalulw 5y ago
    Not always. C++/C are also typically better in memory footprint. You can run more threads in the same server, so better qps.
  - jzoch 5y ago
    Until the data store you grabbed is written in Ruby too :)
  - PragmaticPulp 5y ago
    > That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query. Some of the TechEmpower benchmarks have a significant database component as well. Obviously if you have database queries taking 0.5s or more then it's a different story, but in my experience the server language/framework still has a large impact on total performance even with the database in the equation. In my experience, memory footprint is also a huge variable. For example, I rewrote a certain gRPC service in Rust and cut the memory usage by almost 80% relative to the initial implementation in a higher level language. We weren't bottlenecked on requests/second performance but being able to scale down our memory usage had a huge impact on cloud costs.
  - yen223 5y ago
    There's the Fortunes benchmark, that involves reading off a database: https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r20&hw=... Something to observe from those benchmarks is that "fast" languages are still fast in those benchmarks. The server isn't waiting on I/O 100% of the time - it still has to do serialization and parsing and all other CPU-bound tasks, and those still benefit from a "faster" language.
  - tylerhou 5y ago
    You can hide latency with sufficient parallelism. Since read requests are trivially parallelizable, as long as your database queries are mostly reads the database latency is probably not the limiting factor for throughput. (Even if your C++ program processes too many requests for one replica to handle, it's not hard to round-robin to multiple database replicas.)
  - ksec 5y ago
    Filtered in Fortunes Cookies, Full Stack, ORM Only, MySQL / Postgres DB, and Realistic Approach, the difference is still 50x. That is already lower than the RAW SQL approach from 100 to 150x. And if we just look at .Net and ASP, these were done before all the optimisation came in latest .Net and ASP.net version.
  - ezekiel68 5y ago
    This is a good point, and there are ways to take advantage of this. The c++ frameworks that beat rust advertise in their repos that they are using a forked PostgreSQL DB driver which has been re-written to be asynchronous. There's a js framework named just-js which tops the current (nightly) DB Update benchmark on TechEmpower and I suspect they are doing the same. Additionally, Java, python, and ruby frameworks tend to gobble RAM (I work in a large corp with many deployed Java web services). This requires instances to autoscale based on mem usage or to use instances with more mem. Rust and c++ frameworks often utilize a fraction of the memory for the same service (yes, we've tested this). So we can use smaller and/or fewer tasks/instances in our autoscaling pools.
- derbOac 5y ago
  I tend to think of these things in terms of some limit approaching reality, whatever that is, as the benchmarks continue. If you have an open competition, and there's enough editing and revision by enough actors over time, it gives you some idea of how fast something is likely to be in practice. That open scrutiny and process of suggested revision by experts in a language is a rough sampling of what is likely to be encountered in practice, with a bunch of experts chipping away at the problem. If a difference in times exists, that's about as good as you're likely to get, with further time differences decreasing in size and probability as the whole process continues. I'd even probably argue that even if you have a bunch of naive implementations, it still tells you about what's likely to be encountered in practice, because those naive implementations are often not so naive, and many programmers are more naive than you would like to be the case. Finally, I think in a lot of cases, it's clearer how to do a certain benchmark in a certain language than the essay assumes. Sure, a complex one probably might require some unusual level of expertise. But many benchmarks are fairly small and circumscribed, for better or worse, and consequently pretty tractable.
- lmm 5y ago
  > If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away You're almost certainly not. You almost certainly are nowhere near the RPS limit. You almost certainly are bottlenecked on your own code far more than your framework/language. You're almost certainly running at a performance level of "just barely not bad enough to spend time improving" and would almost certainly be doing the same in any other language. You're probably using a slower framework than the fastest python or ruby framework, so the fact that faster frameworks exist in your language is irrelevant. And if you are in the (rare) case where you're handling enough requests for this choice to matter, then you're big enough to benchmark some proper prototypes that measure the kind of thing your application actually needs to do. Almost anything is a better use of your time than looking at the benchmarks. And if you are going to look at the benchmarks, you shouldn't look at the RPS total; what you should do is first estimate how many requests your app needs to handle and then apply that as a yes/no filter to the frameworks you're considering. "It can handle 1000x as many requests as we need instead of 100x" is not a good reason to pick a framework, much less a language.
  - ezekiel68 5y ago
    I wish this were true -- but it isn't. Partly because Java and scripting frameworks can easily get memory bound instead of CPU bound. I've worked for several large enterprises running externally accessible services in the cloud (and this is true of my current employer). Of course, all of our services have at least 2 "tasks" or instances to help with fault tolerance, yet most of them do, in fact, auto-scale up to more instances during a day based on either RAM or CPU load. (for the curious, apps which read from message queues tend to get CPU bound while web services in Java or a scripting language tend to get memory bound) If you must run 3 instances instead of 2, that is 50% more instance costs for that time period, etc. Moving a service from Java 11 to rust brought the memory load down sufficiently that we didn't need to autoscale at all. Some other service might autoscale, but to fewer max instances. And this adds up multiplied by days by services.
    - lmm 5y ago
      It takes a pretty extreme level of fine tuning to hit that window - Java memory usage is generally about 2x equivalent Rust. (It's completely possible your rewrite saved much more than that - but in that case, doing the same rewrite in Java would probably have brought you similar gains). And unless your user number is very static, you're not going to be in that window for long - maybe you push back the point at which you have to implement autoscaling / do some serious profiling work / start shopping around cloud vendors / ... by six months, or a year, which is helpful but not a huge return on rewriting your whole system. A factor of 10x or 100x is maybe worth letting it determine your choice of language. But given that most performance differences come from algorithm selection rather than the low-level details of your runtime, a language that makes it easy to understand what your code is doing is definitely worth giving up a factor of 2x or even 5x for. For a mature codebase where your userbase is not really growing any more, that kind of marginal performance work can be worthwhile - maybe rather than putting off the point at which you have to scale out, it lets you avoid having to scale out at all and saves you 50% on your running costs forever. But that kind of environment is the same kind of environment where it makes sense to put the codebase into maintenance mode and get rid of most of the expensive programmers (perhaps replacing them with outsourcing).
- slezyr 5y ago
  > Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. Have you seen the source code for the Django app there and one of the top performers? The Django app is just a simple example any developer would write. While Rust's top performers are full of hacks like checking the N-th letter of the HTTP method, no other person would use it in daily life.
  - maverwa 5y ago
    While what you say is true, I don’t see how this invalidates the thesis of the parent comment. Two examples do not invalidate the general trend there. If, however, all implementations are flawed like that, we are having a different problem, right? But yes, super-optimized cases should be removed or marked as such.
    - slezyr 5y ago
      > If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. No one "throws away money" based on the Tech Empower benchmark. Because many of those optimizations are not free and, often, are not available. Such micro-optimizations are present in every program. This benchmark is just noise at this point and should be used as an argument against any framework/language. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/plugins/SyncPlugin.cc#L21 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
  - ezekiel68 5y ago
    This is a common claim -- but no one has yet provided any evidence to back it up. So I got curious and took a look in the TechEmpower/FrameworkBenchmarks repo up on github. For the top rust frameworks (actix and ntex), the submissions are surprisingly short and easily comprehendable. Those two, at least, don't pull anything particularly tricky other than perhaps caching the initial portion of a successful HTTP response string (they all do that). By the same token, there has not been any evidence I have seen to support the claim that the scripting language frameworks have not also done their best with their submissions.
- ashtonkem 5y ago
  > If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. It’s worth pointing out that in some cases developer productivity can swamp server cost. It might be reasonable to pay the server bill necessary to develop in Ruby rather than C++, for example. Obviously the relationship between performance and developer productivity isn’t strictly linear.
  - pclmulqdq 5y ago
    I recently wanted to build a serverless application, but as far as I can tell, no serverless platform takes code in C or C++, so I was stuck working in a language I didn't know well. There may be an opportunity for someone to build a serverless platform that uses verified functions in C/C++/Rust, so you can get both performance and convenience.
    - ezekiel68 5y ago
      This used to be harder -- you would set the lambda up for the go language runtime, but send your own statically linked binary with a special name instead of a compiled go artifact. But now AWS offers Custom Runtimes[0]. So you can set up the infra using some convenient tool such as AWS CloudFormation templates, terraform, or ansible, and then deploy your custom runtime c/c++ build from there (again, using ansible or AWS CLI or whatever). If you're using rust, there are now ways to make that work, including plugins for the popular "serverless" (sls) framework to deploy rust builds[1] [0] https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom... [1] https://github.com/awslabs/aws-lambda-rust-runtime https://github.com/awslabs/aws-lambda-rust-runtime
- nonameiguess 5y ago
  This raises an interesting question to me. When you look at high-performance Python libraries, they achieve the performance by just offloading the critical paths to compiled C modules. I remember reading some comparison of aiohttp to other Python frameworks a while back where the author expected aiohttp to be faster since it doesn't need to block on the network, but it wasn't, largely because of how slow it is to parse http requests in Python. If the bottleneck in serving a ton of requests per second is that Python is bad at serializing and deserializing http headers, why has no one just written a package that offloads that part of the library to a compiled C module? There is seemingly nothing preventing maintaining the convenience of Python for implementing your high-level business logic, but moving the bottlenecks to C. That has been exactly what programmers in scientific computing fields have been doing for 30 years and why projects like NumPy exist in the first place. So why do Python web programmers not just do the same thing? Heck, aiohttp already does this for the event loop, which just uses libuv the same as Node does. Both teams recognize that is a critical path component and it would be really slow to implement it in either Python or Javascript.
atomicnumber3 5y ago
So, I know Ruby is not faster than python, and Java is not faster than C++, and so on. But I've been programming for... 15ish years now, professionally for almost 7, and I've yet to be unable to make a program go "fast enough" regardless of the programming language or runtime it's in. And I've had to do some honestly kinda stupid stuff like doing a bunch of CPU-intensive work in python. And I don't think I'm like, any kind of whiz. I don't know all the deep C/C++/jvm compiler magicks (I honestly find that sort of thing kind of tiresome). I've just found that between multiprocessing (like via xargs -P) and multiprocessing (the python module, or equivalent in a different language or threading in a language with real threads and no interpreter lock) and sometimes shelling out to an OS utility (written in C usually) for a bit of hard work tends to topple any problem, and usually with less damage to the code than a ton of deep magicks would inflict and with less damage to the business function than if it were written in the more-performant less-dev-productive language in the first place. I feel like I need to make a disclaimer since I know several friends who would feel MUCH chagrin at my disparagement of the deep C++ compiler magicks. It's cool, and I have immense respect for their practitioners, but I think my personal interest in programming stems from interest in logic (like the formal stuff you'd learn in a philosophy class) and how comp sci intersects that with being able to build really cool shit (in a product sense). So, you'll have to understand me when I say that it's only in extremely specific scenarios (like HFT, where I used to work, which is how I know all these chagrin'd people in the first place...) that these things tend to make a meaningful different in the product outcome.
- leephillips 5y ago
  Sounds like you’re saying that you’ve been able to make any language fast enough where performance is not critical, but that you realize that when it is (you mention HFT, but there is also HPC for science and engineering, a vast area), the slow languages won’t cut it. I can’t disagree, but your comment also seems like a bit of a tautology.
  - naniwaduni 5y ago
    I think GP is implying a stronger point than you're making it out to, that the situations where performance is critical occur "only in extremely specific scenarios", backed by the claim that they've managed to avoid such scenarios for 7~15 years.
smoldesu 5y ago
Please, correct me if I'm wrong: when I invoke "time" on a Unix system (eg. 'time ./a.out'), is the reported execution time calculated from start to finish, or is it calculated from the sum of CPU cycles? Perhaps a silly question, but I'm curious if *nix developers had the foresight to play with this at any point.
- strken 5y ago
  Real is the wallclock execution time, user and sys are the CPU time for userland and kernel respectively.
  - adrian_b 5y ago
    Yes, "real" is the time from start to finish and for a multi-threaded program "user" and "sys" will be the sums of the corresponding times for all threads, so normally "user" will be many times greater than "real" in that case.
- jjav 5y ago
  Here's an easy test which answers the question: % time sleep 1 sleep 1 0.00s user 0.01s system 0% cpu 1.011 total
prestonbriggs 5y ago
I argued for years in favor of keeping the minimum runtime of a number of samples; never found much support.
choeger 5y ago
What the author talks about is the theoretical performance of a programming language and I honestly believe that sample programs are the wrong experiment for that. My hypothesis is that every programming language can approach optimal performance with enough effort. So the interesting question would be to measure the effort. As a proxy, one could maybe measure the time and experience-level of the programmers. If I am really interested in the theoretical performance of a language, I'd use microbenchmarks for standard library functions. How much time does it take to sort an array? Reverse a list? Split a string? Etc. Then there is of course the performance of the compiler or garbage collector. To compare this between different languages is the most difficult, I think. The best approach here is probably to take some real-world program (say a web server or a database), translate it nearly 1:1 and then compare the results. That's a tremendous effort for a single data point, though (albeit it would yield a nice collection of re-implementations, that could only help software quality).
- Thiez 5y ago
  > If I am really interested in the theoretical performance of a language, I'd use microbenchmarks for standard library functions. How much time does it take to sort an array? Reverse a list? Split a string? Etc. The standard library may not be implemented in the language you're evaluating. Especially for scripting languages, and things like regular expressions and bignums.
Rochus 5y ago
> Or, in other words, the performance difference comes somewhere else than the choice of programming language. We're comparing the code generated by a compiler or VM/JIT of a specific programming language, not the programming language itself. I'm amazed that this is even up for discussion in the article. It is clear that even with the same language there can be differences in implementation. Here is e.g. a measurement series I did to compare different Smalltalk implementations (LjSOM is mine): http://software.rochus-keller.ch/are-we-fast-yet_crystal_lua_node_som_pharo_i386_results_2020-12-29.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_lua.... Smalltalk is a tough nut to crack because nearly all statements are closures; the Open Smalltalk VM team invested two decades to achieve a decent performance, but it's still quite behind e.g. V8 or a statically typed AOT compiled language. The Are-we-fast-yet suite (https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/ https://stefan-marr.de/papers/dls-marr-et-al-cross-language-...) gives more credible results than CLBG (referenced in the article) from my point of view because of the stricter rules the implementations have to adhere to (in favour of fair comparisons) and the larger benchmark apps. In return, the implementation gives significantly more work, but it is doable and worth it; here is one of my implementations that helped me significantly improve the compiler (still in progress): https://github.com/rochus-keller/Oberon/tree/master/testcases/Are-we-fast-yet; https://github.com/rochus-keller/Oberon/tree/master/testcase... this also demonstrates that - given the same platform (LuaJIT bytecode) - Oberon+ was implemented with much less effort and a much more performant result than Smalltalk. And the comparison is "most likely not wrong".

42 points by Sindisil 5y ago

The author raises interesting points, but none of them are novel. The general tone here seems to be along the lines of: there are just so many moving variables, no fair comparisons may be fairly made. But I judge such positions agains this heuristic: "But, is it actionable?" Pragmatism demands we at least take a glance at languages, frameworks, and platforms from the perfromance perspective. Here is a practical example: There may be some issues with the Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. c++, js, amd rust dominate the top 20% and the differences in requests-per-second among them, statistically less significant, literally engulf the total rps numbers for the python or ruby rps totals. If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information.

Retric 5y ago
That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query.
- omegalulw 5y ago
  Not always. C++/C are also typically better in memory footprint. You can run more threads in the same server, so better qps.
- jzoch 5y ago
  Until the data store you grabbed is written in Ruby too :)
- PragmaticPulp 5y ago
  > That’s assuming CPU bound workloads. C++ spends just as long as Ruby waiting on a database query. Some of the TechEmpower benchmarks have a significant database component as well. Obviously if you have database queries taking 0.5s or more then it's a different story, but in my experience the server language/framework still has a large impact on total performance even with the database in the equation. In my experience, memory footprint is also a huge variable. For example, I rewrote a certain gRPC service in Rust and cut the memory usage by almost 80% relative to the initial implementation in a higher level language. We weren't bottlenecked on requests/second performance but being able to scale down our memory usage had a huge impact on cloud costs.
- yen223 5y ago
  There's the Fortunes benchmark, that involves reading off a database: https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=data-r20&hw=... Something to observe from those benchmarks is that "fast" languages are still fast in those benchmarks. The server isn't waiting on I/O 100% of the time - it still has to do serialization and parsing and all other CPU-bound tasks, and those still benefit from a "faster" language.
- tylerhou 5y ago
  You can hide latency with sufficient parallelism. Since read requests are trivially parallelizable, as long as your database queries are mostly reads the database latency is probably not the limiting factor for throughput. (Even if your C++ program processes too many requests for one replica to handle, it's not hard to round-robin to multiple database replicas.)
- ksec 5y ago
  Filtered in Fortunes Cookies, Full Stack, ORM Only, MySQL / Postgres DB, and Realistic Approach, the difference is still 50x. That is already lower than the RAW SQL approach from 100 to 150x. And if we just look at .Net and ASP, these were done before all the optimisation came in latest .Net and ASP.net version.
- ezekiel68 5y ago
  This is a good point, and there are ways to take advantage of this. The c++ frameworks that beat rust advertise in their repos that they are using a forked PostgreSQL DB driver which has been re-written to be asynchronous. There's a js framework named just-js which tops the current (nightly) DB Update benchmark on TechEmpower and I suspect they are doing the same. Additionally, Java, python, and ruby frameworks tend to gobble RAM (I work in a large corp with many deployed Java web services). This requires instances to autoscale based on mem usage or to use instances with more mem. Rust and c++ frameworks often utilize a fraction of the memory for the same service (yes, we've tested this). So we can use smaller and/or fewer tasks/instances in our autoscaling pools.
derbOac 5y ago
I tend to think of these things in terms of some limit approaching reality, whatever that is, as the benchmarks continue. If you have an open competition, and there's enough editing and revision by enough actors over time, it gives you some idea of how fast something is likely to be in practice. That open scrutiny and process of suggested revision by experts in a language is a rough sampling of what is likely to be encountered in practice, with a bunch of experts chipping away at the problem. If a difference in times exists, that's about as good as you're likely to get, with further time differences decreasing in size and probability as the whole process continues. I'd even probably argue that even if you have a bunch of naive implementations, it still tells you about what's likely to be encountered in practice, because those naive implementations are often not so naive, and many programmers are more naive than you would like to be the case. Finally, I think in a lot of cases, it's clearer how to do a certain benchmark in a certain language than the essay assumes. Sure, a complex one probably might require some unusual level of expertise. But many benchmarks are fairly small and circumscribed, for better or worse, and consequently pretty tractable.
lmm 5y ago
> If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away You're almost certainly not. You almost certainly are nowhere near the RPS limit. You almost certainly are bottlenecked on your own code far more than your framework/language. You're almost certainly running at a performance level of "just barely not bad enough to spend time improving" and would almost certainly be doing the same in any other language. You're probably using a slower framework than the fastest python or ruby framework, so the fact that faster frameworks exist in your language is irrelevant. And if you are in the (rare) case where you're handling enough requests for this choice to matter, then you're big enough to benchmark some proper prototypes that measure the kind of thing your application actually needs to do. Almost anything is a better use of your time than looking at the benchmarks. And if you are going to look at the benchmarks, you shouldn't look at the RPS total; what you should do is first estimate how many requests your app needs to handle and then apply that as a yes/no filter to the frameworks you're considering. "It can handle 1000x as many requests as we need instead of 100x" is not a good reason to pick a framework, much less a language.
- ezekiel68 5y ago
  I wish this were true -- but it isn't. Partly because Java and scripting frameworks can easily get memory bound instead of CPU bound. I've worked for several large enterprises running externally accessible services in the cloud (and this is true of my current employer). Of course, all of our services have at least 2 "tasks" or instances to help with fault tolerance, yet most of them do, in fact, auto-scale up to more instances during a day based on either RAM or CPU load. (for the curious, apps which read from message queues tend to get CPU bound while web services in Java or a scripting language tend to get memory bound) If you must run 3 instances instead of 2, that is 50% more instance costs for that time period, etc. Moving a service from Java 11 to rust brought the memory load down sufficiently that we didn't need to autoscale at all. Some other service might autoscale, but to fewer max instances. And this adds up multiplied by days by services.
  - lmm 5y ago
    It takes a pretty extreme level of fine tuning to hit that window - Java memory usage is generally about 2x equivalent Rust. (It's completely possible your rewrite saved much more than that - but in that case, doing the same rewrite in Java would probably have brought you similar gains). And unless your user number is very static, you're not going to be in that window for long - maybe you push back the point at which you have to implement autoscaling / do some serious profiling work / start shopping around cloud vendors / ... by six months, or a year, which is helpful but not a huge return on rewriting your whole system. A factor of 10x or 100x is maybe worth letting it determine your choice of language. But given that most performance differences come from algorithm selection rather than the low-level details of your runtime, a language that makes it easy to understand what your code is doing is definitely worth giving up a factor of 2x or even 5x for. For a mature codebase where your userbase is not really growing any more, that kind of marginal performance work can be worthwhile - maybe rather than putting off the point at which you have to scale out, it lets you avoid having to scale out at all and saves you 50% on your running costs forever. But that kind of environment is the same kind of environment where it makes sense to put the codebase into maintenance mode and get rid of most of the expensive programmers (perhaps replacing them with outsourcing).
slezyr 5y ago
> Tech Empower web framework benchmarks, but -- across a battery of tests -- one may clearly discern that there are NO python or ruby frameworks in the top 75% performing web frameworks yet plenty of go, Java, and PHP frameworks there. Have you seen the source code for the Django app there and one of the top performers? The Django app is just a simple example any developer would write. While Rust's top performers are full of hacks like checking the N-th letter of the HTTP method, no other person would use it in daily life.
- maverwa 5y ago
  While what you say is true, I don’t see how this invalidates the thesis of the parent comment. Two examples do not invalidate the general trend there. If, however, all implementations are flawed like that, we are having a different problem, right? But yes, super-optimized cases should be removed or marked as such.
  - slezyr 5y ago
    > If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. No one "throws away money" based on the Tech Empower benchmark. Because many of those optimizations are not free and, often, are not available. Such micro-optimizations are present in every program. This benchmark is just noise at this point and should be used as an argument against any framework/language. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/plugins/SyncPlugin.cc#L21 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
- ezekiel68 5y ago
  This is a common claim -- but no one has yet provided any evidence to back it up. So I got curious and took a look in the TechEmpower/FrameworkBenchmarks repo up on github. For the top rust frameworks (actix and ntex), the submissions are surprisingly short and easily comprehendable. Those two, at least, don't pull anything particularly tricky other than perhaps caching the initial portion of a successful HTTP response string (they all do that). By the same token, there has not been any evidence I have seen to support the claim that the scripting language frameworks have not also done their best with their submissions.
ashtonkem 5y ago
> If you (or your company) are paying cloud fees for hosting your services (including serverless), you are absolutely throwing money away, perhaps for a "good" reason ('we have a huge rails codebase'), if you ignore this information. It’s worth pointing out that in some cases developer productivity can swamp server cost. It might be reasonable to pay the server bill necessary to develop in Ruby rather than C++, for example. Obviously the relationship between performance and developer productivity isn’t strictly linear.
- pclmulqdq 5y ago
  I recently wanted to build a serverless application, but as far as I can tell, no serverless platform takes code in C or C++, so I was stuck working in a language I didn't know well. There may be an opportunity for someone to build a serverless platform that uses verified functions in C/C++/Rust, so you can get both performance and convenience.
  - ezekiel68 5y ago
    This used to be harder -- you would set the lambda up for the go language runtime, but send your own statically linked binary with a special name instead of a compiled go artifact. But now AWS offers Custom Runtimes[0]. So you can set up the infra using some convenient tool such as AWS CloudFormation templates, terraform, or ansible, and then deploy your custom runtime c/c++ build from there (again, using ansible or AWS CLI or whatever). If you're using rust, there are now ways to make that work, including plugins for the popular "serverless" (sls) framework to deploy rust builds[1] [0] https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom... [1] https://github.com/awslabs/aws-lambda-rust-runtime https://github.com/awslabs/aws-lambda-rust-runtime
nonameiguess 5y ago
This raises an interesting question to me. When you look at high-performance Python libraries, they achieve the performance by just offloading the critical paths to compiled C modules. I remember reading some comparison of aiohttp to other Python frameworks a while back where the author expected aiohttp to be faster since it doesn't need to block on the network, but it wasn't, largely because of how slow it is to parse http requests in Python. If the bottleneck in serving a ton of requests per second is that Python is bad at serializing and deserializing http headers, why has no one just written a package that offloads that part of the library to a compiled C module? There is seemingly nothing preventing maintaining the convenience of Python for implementing your high-level business logic, but moving the bottlenecks to C. That has been exactly what programmers in scientific computing fields have been doing for 30 years and why projects like NumPy exist in the first place. So why do Python web programmers not just do the same thing? Heck, aiohttp already does this for the event loop, which just uses libuv the same as Node does. Both teams recognize that is a critical path component and it would be really slow to implement it in either Python or Javascript.

atomicnumber3 5y ago

So, I know Ruby is not faster than python, and Java is not faster than C++, and so on. But I've been programming for... 15ish years now, professionally for almost 7, and I've yet to be unable to make a program go "fast enough" regardless of the programming language or runtime it's in. And I've had to do some honestly kinda stupid stuff like doing a bunch of CPU-intensive work in python. And I don't think I'm like, any kind of whiz. I don't know all the deep C/C++/jvm compiler magicks (I honestly find that sort of thing kind of tiresome). I've just found that between multiprocessing (like via xargs -P) and multiprocessing (the python module, or equivalent in a different language or threading in a language with real threads and no interpreter lock) and sometimes shelling out to an OS utility (written in C usually) for a bit of hard work tends to topple any problem, and usually with less damage to the code than a ton of deep magicks would inflict and with less damage to the business function than if it were written in the more-performant less-dev-productive language in the first place. I feel like I need to make a disclaimer since I know several friends who would feel MUCH chagrin at my disparagement of the deep C++ compiler magicks. It's cool, and I have immense respect for their practitioners, but I think my personal interest in programming stems from interest in logic (like the formal stuff you'd learn in a philosophy class) and how comp sci intersects that with being able to build really cool shit (in a product sense). So, you'll have to understand me when I say that it's only in extremely specific scenarios (like HFT, where I used to work, which is how I know all these chagrin'd people in the first place...) that these things tend to make a meaningful different in the product outcome.

leephillips 5y ago
Sounds like you’re saying that you’ve been able to make any language fast enough where performance is not critical, but that you realize that when it is (you mention HFT, but there is also HPC for science and engineering, a vast area), the slow languages won’t cut it. I can’t disagree, but your comment also seems like a bit of a tautology.
- naniwaduni 5y ago
  I think GP is implying a stronger point than you're making it out to, that the situations where performance is critical occur "only in extremely specific scenarios", backed by the claim that they've managed to avoid such scenarios for 7~15 years.

smoldesu 5y ago

Please, correct me if I'm wrong: when I invoke "time" on a Unix system (eg. 'time ./a.out'), is the reported execution time calculated from start to finish, or is it calculated from the sum of CPU cycles? Perhaps a silly question, but I'm curious if *nix developers had the foresight to play with this at any point.

strken 5y ago
Real is the wallclock execution time, user and sys are the CPU time for userland and kernel respectively.
- adrian_b 5y ago
  Yes, "real" is the time from start to finish and for a multi-threaded program "user" and "sys" will be the sums of the corresponding times for all threads, so normally "user" will be many times greater than "real" in that case.
jjav 5y ago
Here's an easy test which answers the question: % time sleep 1 sleep 1 0.00s user 0.01s system 0% cpu 1.011 total

prestonbriggs 5y ago

I argued for years in favor of keeping the minimum runtime of a number of samples; never found much support.

choeger 5y ago

What the author talks about is the theoretical performance of a programming language and I honestly believe that sample programs are the wrong experiment for that. My hypothesis is that every programming language can approach optimal performance with enough effort. So the interesting question would be to measure the effort. As a proxy, one could maybe measure the time and experience-level of the programmers. If I am really interested in the theoretical performance of a language, I'd use microbenchmarks for standard library functions. How much time does it take to sort an array? Reverse a list? Split a string? Etc. Then there is of course the performance of the compiler or garbage collector. To compare this between different languages is the most difficult, I think. The best approach here is probably to take some real-world program (say a web server or a database), translate it nearly 1:1 and then compare the results. That's a tremendous effort for a single data point, though (albeit it would yield a nice collection of re-implementations, that could only help software quality).

Thiez 5y ago
> If I am really interested in the theoretical performance of a language, I'd use microbenchmarks for standard library functions. How much time does it take to sort an array? Reverse a list? Split a string? Etc. The standard library may not be implemented in the language you're evaluating. Especially for scripting languages, and things like regular expressions and bignums.

Rochus 5y ago

> Or, in other words, the performance difference comes somewhere else than the choice of programming language. We're comparing the code generated by a compiler or VM/JIT of a specific programming language, not the programming language itself. I'm amazed that this is even up for discussion in the article. It is clear that even with the same language there can be differences in implementation. Here is e.g. a measurement series I did to compare different Smalltalk implementations (LjSOM is mine): http://software.rochus-keller.ch/are-we-fast-yet_crystal_lua_node_som_pharo_i386_results_2020-12-29.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_lua.... Smalltalk is a tough nut to crack because nearly all statements are closures; the Open Smalltalk VM team invested two decades to achieve a decent performance, but it's still quite behind e.g. V8 or a statically typed AOT compiled language. The Are-we-fast-yet suite (https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/ https://stefan-marr.de/papers/dls-marr-et-al-cross-language-...) gives more credible results than CLBG (referenced in the article) from my point of view because of the stricter rules the implementations have to adhere to (in favour of fair comparisons) and the larger benchmark apps. In return, the implementation gives significantly more work, but it is doable and worth it; here is one of my implementations that helped me significantly improve the compiler (still in progress): https://github.com/rochus-keller/Oberon/tree/master/testcases/Are-we-fast-yet; https://github.com/rochus-keller/Oberon/tree/master/testcase... this also demonstrates that - given the same platform (LuaJIT bytecode) - Oberon+ was implemented with much less effort and a much more performant result than Smalltalk. And the comparison is "most likely not wrong".