7 ms·

Perl vs Scala vs Go vs C (performance)

9 points by systems 16y ago

_delirium 16y ago
Although these are important real-world cases, I think this kind of high-throughput file processing mostly benchmarks something other than the core language design. More or less: 1) how good are the I/O libraries and string-processing libraries?; and 2) how obvious is it which ones to use, and how to use them, for peak performance in your use case? As he noted, he got a 4x speedup in Scala by just shuffling around his use of the Scala library functions, so this sort of benchmarking is quite sensitive to seemingly minor choices. This kind of I/O benchmarking is also more sensitive than most to the particular choice of hardware. Results might come out totally differently on a different kind of hard drive, 32-bit v. 64-bit, different OS or filesystem, etc. The various entries in the Wide Finder competition give a good overview of the kinds of tricks that can make a difference when you throw another wrench in the works by combining high-throughput I/O with multiprocessing (some of the tweaks aren't multiprocessing-specific, though): http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finde...
- barrkel 16y ago
  Indeed. And C gives you such crummy tools for string manipulation that you are almost forced to hand-write anything interesting, so it's not surprising that it ends up doing well.
  - _delirium 16y ago
    That's one way to avoid having slow or misused library functions kill you in benchmarks-- don't give the user many library functions. ;-)
  - sandGorgon 16y ago
    What about Boost strings - http://www.boost.org/doc/libs/1_39_0/doc/html/string_algo.html http://www.boost.org/doc/libs/1_39_0/doc/html/string_algo.ht... and RE2 for regex - http://code.google.com/p/re2/ http://code.google.com/p/re2/
    - _delirium 16y ago
      Those are both C++, not C, aren't they?
  - JoachimSchipper 16y ago
    If your non-crummy tools are so bad that C ends up eating your lunch by a factor of 100, I'm not sure that they really are non-crummy.
    - barrkel 16y ago
      Powerful tools improve productivity because you have to do less work to get the job accomplished. But they have a reverse side: you need to be aware of how they work to use them efficiently, otherwise you'll end up with silly performance problems - e.g. I've seen naive string tokenization code that did substrings, and then deleted the matched substring from the start of the string, all in a loop. By providing the tools, it's easier to misuse them. When you have fewer tools, on the other hand, you need to think at a lower level and work more directly on the problem. You have a lot more work to do, but the probability that you're misusing the tools is smaller, because you'll be building the tools yourself, manually.
      - _delirium 16y ago
        I wonder how much that actually accounts for C's generally good performance in these kinds of ad-hoc benchmarks. There's really two "low-level" things about C: 1) it's low-level in the actual language sense of being reasonably close to the metal; and 2) it has a pretty bare-bones standard library, requiring most algorithms to be hand-coded, if you don't link in a 3rd-party library. Mostly people focus on #1, but as you point out, #2 might actually be a big part of it. I don't actually mean this as a slight against C, fwiw. In some ways, coding in C feels liberating, because I'm culturally allowed to do the nasty special-case hack, instead of doing it The Right Way with the library function that checks all the edge cases and error conditions and whatever.
        barrkel 16y ago
        On this theme, there was a kind of "code-off" between Rico Mariani (.NET performance architect at the time) and Raymond Chen (The Old New Thing blog, Windows user mode developer). Rico has a kind of index to it here: http://blogs.msdn.com/ricom/archive/2005/05/10/performance-quiz-6-chinese-english-dictionary-reader.aspx http://blogs.msdn.com/ricom/archive/2005/05/10/performance-q... It was interesting to see how little Rico had to do from the start to end up with quite a performant application. Eventually Raymond started pulling ahead, but IIRC it was when he started using memory mapped files for I/O. Here's a summary at version 5 of the unmanaged code: http://blogs.msdn.com/ricom/archive/2005/05/18/419805.aspx http://blogs.msdn.com/ricom/archive/2005/05/18/419805.aspx Version Execution Time (seconds) Unmanaged v1 1.328 Unmanaged v2 0.828 Unmanaged v3 0.343 Unmanaged v4 0.187 Unmanaged v5 With Bug 0.296 Unmanaged v5 Corrected 0.124 Unoptimized Managed port of v1 0.124 It was quite interesting.
JoachimSchipper 16y ago
As could be expected, this just benchmarks the implementation. For instance, Perl has lots of object creation while C uses a fast state-machine-like approach. Don't get me wrong - I really like C - but I'm not even sure that it would still be faster than Perl if you switched implementations.
- fleitz 16y ago
  Agreed. Sure in scala there are a couple things you can't do like pointer manipulation, so you are going to incur more allocations but from what I've found in C# is, if you code it like a C app it performs like a C app. The biggest key to performance I've found in managed code is to not allocate memory. It reduces stress on the GC, and improves your cache hit rate.
  - barrkel 16y ago
    I can't agree with the non-allocation of memory. What you don't want to do is create garbage that doesn't die until after a collection has already occurred - this is the mid-life crisis problem. But you shouldn't be unduly afraid of allocating memory which won't last long. I wrote up a quick test sample: http://pastebin.com/YmbT08Uy http://pastebin.com/YmbT08Uy All it does is fill in an array with values from 0 to 511, then sum them up. The catch is that it zeros out the array before filling in the values, in two different ways: one (P), by re-using an array, and secondly (Q) by reallocating the array and letting the garbage collector zero it out. Here are test run timings: P: 8.749 Q: 7.070 Result: 0 And in reversed order, to show that order of compilation didn't make a difference: Q: 7.081 P: 8.746 Result: 0 The code using the GC to implicitly zero out the array is still faster than the code that reuses the array, even though the GC is having to collect a large fraction of the 10 million 2048-byte arrays allocated by Q. P, on the other hand, only allocates 100 of them. Test machine: i7 920 @ 3.6GHz, 12GB memory, .NET 4 on Win7 64-bit. Don't forget that time spent in GC is proportional to the amount of live data in the generation(s) being collected, not the amount of garbage. Collecting garbage is actually free; it's finding out what's garbage that costs. If you don't keep references around to memory (i.e. increase your live set), GC is very cheap.
AlisdairO 16y ago
Interesting, but not unexpected results for Scala. The JVM takes a while to start up, so on short lived scripting-like applications it's not an ideal choice.
- runT1ME 16y ago
  Startup and to JIT the code. I bet if they accounted for that, we'd see Scala even more performant.
fleitz 16y ago
Language performance benchmarks are generally pointless. The much more important benchmark is how easy is the code to write / change. Write it quick, get the customer, and then when you have that customers money if somehow it doesn't perform well enough use that money to profile and improve the app or just buy more hardware. In 18 months a CPU that costs $200 will perform twice as fast as the current CPU for $200, I've never seen bang for the buck like doubling the performance of your code for $200. (Most apps are IO bound anyway, so you want to add disks not CPU) Yes, for SOME companies like google it's much better to pay more for programmers, but most companies don't have > 100,000 servers. If your company is a start up that is doubling every year then all you have to do is let Moore's law scale your performance as you add servers. Also, don't build a business where your profit margin is dependent on getting the last 5% performance out of a server.
- barrkel 16y ago
  It very much depends on your application domain. For code that runs on client devices, speed is often very important. You can't (usually) buy hardware for your customers.
- Legion 16y ago
  I think "generally pointless" is overstating things a bit. That said, I think it is definitely fair to say that these sort of benchmarks fall more into the category of "interesting" than "useful". Usually fun to read, but just not something to confuse with information to base business decisions on.
- runT1ME 16y ago
  >In 18 months a CPU that costs $200 will perform twice as fast as the current CPU for $200, I've never seen bang for the buck like doubling the performance of your code for $200. (Most apps are IO bound anyway, so you want to add disks not CPU) 2001 called, they want their answer back. Ok, that's a little harsh, but really, you're answer was perfect for about six years ago, but if you've taken a look at chip performance recently, it has certainly not been doubling every year. What you can get for cheaper is more cores, but that certainly isn't a free lunch, as many have pointed out. Suddenly having a good language that can be per formant and scale matters (python anyone?). Go is going to have a lot of catching up to do, as writing good multi threaded libraries are hard, and a good parallel GC is even harder. I don't mean to pick on your opinion, because its one shared by many, but things are going to get very interesting in the next 10 years as we really will see the end of moore's law in terms of linear CPU performance and instead get many-core processors to play with.
gjm11 16y ago
The most surprising-looking thing here is the really wretched results from the Go version. But when you actually take a look at the Go code, that stops being so surprising; he's using a CSV-reading library whose author clearly wasn't thinking much about performance. It doesn't contain anything really super-dreadful -- O(n^2) algorithms for processing strings, etc. -- but it's full of gratuitous constant-factor inefficiencies compared with what you'd be likely to do in, say, C. Unnecessary temporary objects (presumably allocated on the GCed heap), building short strings by { read a byte, push it into a string-builder, repeat }, etc. It's the kind of code is makes sense to write in a higher-level language where (1) allocation overhead is likely to be swamped by interpretation overhead and (2) you know that for anything that really needs to be fast you'll punt to an extension library written in C. But Go is not such a language, even though it tries to offer some of the same conveniences as they do.
- runT1ME 16y ago
  Go uses a primitive garbage collector also, so while Scala can create boatloads of extraneous objects, their allocation costs are negligible and their collection cost is 0.
- deleted 16y ago
  [deleted]
tumult 16y ago
This is really stupid.
acqq 16y ago
It seems that the author doesn't have the real idea what he actually wants to measure? He just throws some heaps of code and gives some numbers. I haven't seen that he even tried to specify what he wants to achieve. His posts just don't have sense. Something which is really useful is on: http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/ For example: http://shootout.alioth.debian.org/u64q/compare.php?lang=go http://shootout.alioth.debian.org/u64q/compare.php?lang=go etc

9 points by systems 16y ago

Although these are important real-world cases, I think this kind of high-throughput file processing mostly benchmarks something other than the core language design. More or less: 1) how good are the I/O libraries and string-processing libraries?; and 2) how obvious is it which ones to use, and how to use them, for peak performance in your use case? As he noted, he got a 4x speedup in Scala by just shuffling around his use of the Scala library functions, so this sort of benchmarking is quite sensitive to seemingly minor choices. This kind of I/O benchmarking is also more sensitive than most to the particular choice of hardware. Results might come out totally differently on a different kind of hard drive, 32-bit v. 64-bit, different OS or filesystem, etc. The various entries in the Wide Finder competition give a good overview of the kinds of tricks that can make a difference when you throw another wrench in the works by combining high-throughput I/O with multiprocessing (some of the tweaks aren't multiprocessing-specific, though): http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finde...

barrkel 16y ago
Indeed. And C gives you such crummy tools for string manipulation that you are almost forced to hand-write anything interesting, so it's not surprising that it ends up doing well.
- _delirium 16y ago
  That's one way to avoid having slow or misused library functions kill you in benchmarks-- don't give the user many library functions. ;-)
- sandGorgon 16y ago
  What about Boost strings - http://www.boost.org/doc/libs/1_39_0/doc/html/string_algo.html http://www.boost.org/doc/libs/1_39_0/doc/html/string_algo.ht... and RE2 for regex - http://code.google.com/p/re2/ http://code.google.com/p/re2/
  - _delirium 16y ago
    Those are both C++, not C, aren't they?
- JoachimSchipper 16y ago
  If your non-crummy tools are so bad that C ends up eating your lunch by a factor of 100, I'm not sure that they really are non-crummy.
  - barrkel 16y ago
    Powerful tools improve productivity because you have to do less work to get the job accomplished. But they have a reverse side: you need to be aware of how they work to use them efficiently, otherwise you'll end up with silly performance problems - e.g. I've seen naive string tokenization code that did substrings, and then deleted the matched substring from the start of the string, all in a loop. By providing the tools, it's easier to misuse them. When you have fewer tools, on the other hand, you need to think at a lower level and work more directly on the problem. You have a lot more work to do, but the probability that you're misusing the tools is smaller, because you'll be building the tools yourself, manually.
    - _delirium 16y ago
      I wonder how much that actually accounts for C's generally good performance in these kinds of ad-hoc benchmarks. There's really two "low-level" things about C: 1) it's low-level in the actual language sense of being reasonably close to the metal; and 2) it has a pretty bare-bones standard library, requiring most algorithms to be hand-coded, if you don't link in a 3rd-party library. Mostly people focus on #1, but as you point out, #2 might actually be a big part of it. I don't actually mean this as a slight against C, fwiw. In some ways, coding in C feels liberating, because I'm culturally allowed to do the nasty special-case hack, instead of doing it The Right Way with the library function that checks all the edge cases and error conditions and whatever.
      - barrkel 16y ago
        On this theme, there was a kind of "code-off" between Rico Mariani (.NET performance architect at the time) and Raymond Chen (The Old New Thing blog, Windows user mode developer). Rico has a kind of index to it here: http://blogs.msdn.com/ricom/archive/2005/05/10/performance-quiz-6-chinese-english-dictionary-reader.aspx http://blogs.msdn.com/ricom/archive/2005/05/10/performance-q... It was interesting to see how little Rico had to do from the start to end up with quite a performant application. Eventually Raymond started pulling ahead, but IIRC it was when he started using memory mapped files for I/O. Here's a summary at version 5 of the unmanaged code: http://blogs.msdn.com/ricom/archive/2005/05/18/419805.aspx http://blogs.msdn.com/ricom/archive/2005/05/18/419805.aspx Version Execution Time (seconds) Unmanaged v1 1.328 Unmanaged v2 0.828 Unmanaged v3 0.343 Unmanaged v4 0.187 Unmanaged v5 With Bug 0.296 Unmanaged v5 Corrected 0.124 Unoptimized Managed port of v1 0.124 It was quite interesting.

JoachimSchipper 16y ago

As could be expected, this just benchmarks the implementation. For instance, Perl has lots of object creation while C uses a fast state-machine-like approach. Don't get me wrong - I really like C - but I'm not even sure that it would still be faster than Perl if you switched implementations.

fleitz 16y ago
Agreed. Sure in scala there are a couple things you can't do like pointer manipulation, so you are going to incur more allocations but from what I've found in C# is, if you code it like a C app it performs like a C app. The biggest key to performance I've found in managed code is to not allocate memory. It reduces stress on the GC, and improves your cache hit rate.
- barrkel 16y ago
  I can't agree with the non-allocation of memory. What you don't want to do is create garbage that doesn't die until after a collection has already occurred - this is the mid-life crisis problem. But you shouldn't be unduly afraid of allocating memory which won't last long. I wrote up a quick test sample: http://pastebin.com/YmbT08Uy http://pastebin.com/YmbT08Uy All it does is fill in an array with values from 0 to 511, then sum them up. The catch is that it zeros out the array before filling in the values, in two different ways: one (P), by re-using an array, and secondly (Q) by reallocating the array and letting the garbage collector zero it out. Here are test run timings: P: 8.749 Q: 7.070 Result: 0 And in reversed order, to show that order of compilation didn't make a difference: Q: 7.081 P: 8.746 Result: 0 The code using the GC to implicitly zero out the array is still faster than the code that reuses the array, even though the GC is having to collect a large fraction of the 10 million 2048-byte arrays allocated by Q. P, on the other hand, only allocates 100 of them. Test machine: i7 920 @ 3.6GHz, 12GB memory, .NET 4 on Win7 64-bit. Don't forget that time spent in GC is proportional to the amount of live data in the generation(s) being collected, not the amount of garbage. Collecting garbage is actually free; it's finding out what's garbage that costs. If you don't keep references around to memory (i.e. increase your live set), GC is very cheap.

AlisdairO 16y ago

Interesting, but not unexpected results for Scala. The JVM takes a while to start up, so on short lived scripting-like applications it's not an ideal choice.

runT1ME 16y ago
Startup and to JIT the code. I bet if they accounted for that, we'd see Scala even more performant.

fleitz 16y ago

Language performance benchmarks are generally pointless. The much more important benchmark is how easy is the code to write / change. Write it quick, get the customer, and then when you have that customers money if somehow it doesn't perform well enough use that money to profile and improve the app or just buy more hardware. In 18 months a CPU that costs $200 will perform twice as fast as the current CPU for $200, I've never seen bang for the buck like doubling the performance of your code for $200. (Most apps are IO bound anyway, so you want to add disks not CPU) Yes, for SOME companies like google it's much better to pay more for programmers, but most companies don't have > 100,000 servers. If your company is a start up that is doubling every year then all you have to do is let Moore's law scale your performance as you add servers. Also, don't build a business where your profit margin is dependent on getting the last 5% performance out of a server.

barrkel 16y ago
It very much depends on your application domain. For code that runs on client devices, speed is often very important. You can't (usually) buy hardware for your customers.
Legion 16y ago
I think "generally pointless" is overstating things a bit. That said, I think it is definitely fair to say that these sort of benchmarks fall more into the category of "interesting" than "useful". Usually fun to read, but just not something to confuse with information to base business decisions on.
runT1ME 16y ago
>In 18 months a CPU that costs $200 will perform twice as fast as the current CPU for $200, I've never seen bang for the buck like doubling the performance of your code for $200. (Most apps are IO bound anyway, so you want to add disks not CPU) 2001 called, they want their answer back. Ok, that's a little harsh, but really, you're answer was perfect for about six years ago, but if you've taken a look at chip performance recently, it has certainly not been doubling every year. What you can get for cheaper is more cores, but that certainly isn't a free lunch, as many have pointed out. Suddenly having a good language that can be per formant and scale matters (python anyone?). Go is going to have a lot of catching up to do, as writing good multi threaded libraries are hard, and a good parallel GC is even harder. I don't mean to pick on your opinion, because its one shared by many, but things are going to get very interesting in the next 10 years as we really will see the end of moore's law in terms of linear CPU performance and instead get many-core processors to play with.

gjm11 16y ago

The most surprising-looking thing here is the really wretched results from the Go version. But when you actually take a look at the Go code, that stops being so surprising; he's using a CSV-reading library whose author clearly wasn't thinking much about performance. It doesn't contain anything really super-dreadful -- O(n^2) algorithms for processing strings, etc. -- but it's full of gratuitous constant-factor inefficiencies compared with what you'd be likely to do in, say, C. Unnecessary temporary objects (presumably allocated on the GCed heap), building short strings by { read a byte, push it into a string-builder, repeat }, etc. It's the kind of code is makes sense to write in a higher-level language where (1) allocation overhead is likely to be swamped by interpretation overhead and (2) you know that for anything that really needs to be fast you'll punt to an extension library written in C. But Go is not such a language, even though it tries to offer some of the same conveniences as they do.

runT1ME 16y ago
Go uses a primitive garbage collector also, so while Scala can create boatloads of extraneous objects, their allocation costs are negligible and their collection cost is 0.
deleted 16y ago
[deleted]

tumult 16y ago

This is really stupid.

acqq 16y ago

It seems that the author doesn't have the real idea what he actually wants to measure? He just throws some heaps of code and gives some numbers. I haven't seen that he even tried to specify what he wants to achieve. His posts just don't have sense. Something which is really useful is on: http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/ For example: http://shootout.alioth.debian.org/u64q/compare.php?lang=go http://shootout.alioth.debian.org/u64q/compare.php?lang=go etc