6 ms·

Go vs. Swift vs. C++ microbenchmark

63 points by monort 11y ago

marvel_boy 11y ago
I'd love to see Erlang here. Anyway Swift performance is impressive.
DannyBee 11y ago
Oh microbenchmarks. All of this is going to be optimized away by a sufficiently smart compiler or a compiler you tell you want to spend lots of time optimizing. In general: don't write microbenchmarks where the results can be easily statically evaluated by the compiler, unless you are testing whether the compiler can statically evaluate it (this is a reasonable thing to want to know of course, but it's not a test of the language). There are also languages which will pretty much not bother to do a lot of the the actual work, like haskell. (Note: In this case, the the C++ compiler being used actually knows all the recurrences involved, but decides it's too expensive to statically calculate the final result of sum. GCC would do the same.)
- 21 11y ago
  Doesn't the std::vector calls mess things up? Now the compiler has to track those as well, they involve the allocator which calls into the OS. So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects. Of course, the compiler could have special knowledge of std::vector, like for example in general it has about memcpy.
  - DannyBee 11y ago
    "So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects. " It can know this the same way it knows about memcpy. Most of them have library call info for things that have standardized guarantees. Even if it didn't, it knows what malloc does, it knows this variable never escapes, and ...
- pjmlp 11y ago
  Not only that, usually all the ones applied via gut feeling instead of a profiler are meaninglessness. Additionally, depending on the application domain, it might not even matter.
zeckalpha 11y ago
And Haskell, in the comments.
stepanhruda 11y ago
Concurrency model will be a big focus for Swift 4. (Swift is currently v2.2, the focus of v3 is to stabilize the ABI.)
campoy 11y ago
I don't care about micro benchmarks ... but if you're gonna do one, make sure your code is decent. By replacing the Go slice declarations (var x[]int64) with (x := make([]int64, 0, 1000000)) you'll get an algorithm that in my iMac runs much faster: - your version: 4.93s user 0.28s system 165% cpu 3.151 total - my version: 1.93s user 0.04s system 168% cpu 1.165 total EDIT: I'm sure this happens with all the examples, and that's exactly my point. What is this exactly microbenchmarking? Bad programs?
- masklinn 11y ago
  (Re)allocating arrays/vectors seems to be pretty much the whole point, otherwise both Swift and C++ could not only preallocate (reserve), but they could use iterators and elide all allocation instead, and then you'd get something along the lines of this: > rustc -O test.rs -o test_rs > time ./test_rs 9999010000 0.54 real 0.53 user 0.00 sys here's TFA's C++ version by comparison: > time ./test_cpp 9999010000 3.17 real 2.29 user 0.86 sys
  - campoy 11y ago
    I totally agree with you on this. The code for all three languages is clearly not optimal. I wonder how much of this performance difference is C++ being very good at optimizing your code rather than the runtime being more performant. I don't have any numbers, but I feel like that's not a negligible factor.
    - masklinn 11y ago
      > I totally agree with you on this. The code for all three languages is clearly not optimal. That's not really my point. My point's we have no idea what TFA is trying to do or see[0], so you may be able to say "this is stupid and nonsensical" because it is[1], but you can't say "I've got a better version" then provide something with a completely different behaviour. [0] most likely they just threw some shit at the wall, ended up with relatively long runtimes and called that a benchmark [1] or maybe not, maybe it's a reduction of actual workload in the system doing computation based on some sort of dynamic number of items so you can't preallocate the intermediate storage and the point's to compare reallocation overhead[2] [2] not very likely though, considering you never need the array in the first place as you're just summing each item with the previous one, then summing every 100 item of that… anyway
rurban 11y ago
If you want to compare fast languages, you need to compare C++ with rust and pony, not swift and go.
- pjmlp 11y ago
  And include multiple implementations of each one, as that is what really counts.
mangahero 11y ago
How sad. You have to be equivalent for all the languages. Mr. Twitter Developer Advocate at Google
- campoy 11y ago
  I know, and that is exactly my point. Micro benchmarks across languages with code that doesn't do anything interesting is pretty pointless IMHO. Also, am I a twitter developer advocate now? what is that?
  - zaius 11y ago
    I'm guessing s/he's misreading your bio line in your profile.
- dang 11y ago
  Personal attacks (which that looks like) are not allowed on Hacker News. We detached this subthread from https://news.ycombinator.com/item?id=11098802 https://news.ycombinator.com/item?id=11098802 and marked it off-topic.
oussama-gmd 11y ago
Where is Rust :)
- jbeja 11y ago
  In academia.
  - bluejekyll 11y ago
    This is totally unfair. We're looking at using Rust for production workloads and cross system development.
justinsb 11y ago
As far as micro-benchmarks go, this one is particularly egregious. It is likely completely dominated by the array reallocation. So a runtime that - say - chooses to preallocate an array at size 1000000, or to go to 1000000 at first resize would win at this benchmark. More realistically, a runtime that conserves memory by reallocating arrays with a 1.5x factor vs a 2x factor will appear slower. It's exactly because you don't want to rely on the runtime making the right call for your use-case that you preallocate arrays where it makes a difference (i.e. where you're appending a million items). And 99% of the time you're doing less than 100 items, and so the array reallocation behavior just doesn't matter.
0x434D53 11y ago
So what's the conclusion of this besides some useless numbers?
jbeja 11y ago
What's the point, seriously?
otec 11y ago
Do you actually understand what are you measuring in C++ example ? I dont think you have sufficient knowledge to make any benchmarks.
autoreleasepool 11y ago
FYI, unary operators and C-style for loops have been depreciated in Swift.
jsksma2 11y ago
> "But it adds some checks and GC, so Swift also exhibits a serious slowdown, though well within 2x of C++." Swift does not employ GC, it uses ARC. While that may be a form of GC, it doesn't suffer from the same "slowdown" that GC would in this context because ARC is doing exactly what your base case (C++) does at run-time (ARC is a compile-time tool). I think if you're going to be doing benchmarks, you should have basic knowledge on how each system works first.
chvid 11y ago
I really think this is mostly just testing how well the list/vector/array class is handling being asked to expand itself by one element a million times.

63 points by monort 11y ago

I'd love to see Erlang here. Anyway Swift performance is impressive.

Oh microbenchmarks. All of this is going to be optimized away by a sufficiently smart compiler or a compiler you tell you want to spend lots of time optimizing. In general: don't write microbenchmarks where the results can be easily statically evaluated by the compiler, unless you are testing whether the compiler can statically evaluate it (this is a reasonable thing to want to know of course, but it's not a test of the language). There are also languages which will pretty much not bother to do a lot of the the actual work, like haskell. (Note: In this case, the the C++ compiler being used actually knows all the recurrences involved, but decides it's too expensive to statically calculate the final result of sum. GCC would do the same.)

21 11y ago
Doesn't the std::vector calls mess things up? Now the compiler has to track those as well, they involve the allocator which calls into the OS. So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects. Of course, the compiler could have special knowledge of std::vector, like for example in general it has about memcpy.
- DannyBee 11y ago
  "So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects. " It can know this the same way it knows about memcpy. Most of them have library call info for things that have standardized guarantees. Even if it didn't, it knows what malloc does, it knows this variable never escapes, and ...
pjmlp 11y ago
Not only that, usually all the ones applied via gut feeling instead of a profiler are meaninglessness. Additionally, depending on the application domain, it might not even matter.

zeckalpha 11y ago

And Haskell, in the comments.

stepanhruda 11y ago

Concurrency model will be a big focus for Swift 4. (Swift is currently v2.2, the focus of v3 is to stabilize the ABI.)

campoy 11y ago

I don't care about micro benchmarks ... but if you're gonna do one, make sure your code is decent. By replacing the Go slice declarations (var x[]int64) with (x := make([]int64, 0, 1000000)) you'll get an algorithm that in my iMac runs much faster: - your version: 4.93s user 0.28s system 165% cpu 3.151 total - my version: 1.93s user 0.04s system 168% cpu 1.165 total EDIT: I'm sure this happens with all the examples, and that's exactly my point. What is this exactly microbenchmarking? Bad programs?

masklinn 11y ago
(Re)allocating arrays/vectors seems to be pretty much the whole point, otherwise both Swift and C++ could not only preallocate (reserve), but they could use iterators and elide all allocation instead, and then you'd get something along the lines of this: > rustc -O test.rs -o test_rs > time ./test_rs 9999010000 0.54 real 0.53 user 0.00 sys here's TFA's C++ version by comparison: > time ./test_cpp 9999010000 3.17 real 2.29 user 0.86 sys
- campoy 11y ago
  I totally agree with you on this. The code for all three languages is clearly not optimal. I wonder how much of this performance difference is C++ being very good at optimizing your code rather than the runtime being more performant. I don't have any numbers, but I feel like that's not a negligible factor.
  - masklinn 11y ago
    > I totally agree with you on this. The code for all three languages is clearly not optimal. That's not really my point. My point's we have no idea what TFA is trying to do or see[0], so you may be able to say "this is stupid and nonsensical" because it is[1], but you can't say "I've got a better version" then provide something with a completely different behaviour. [0] most likely they just threw some shit at the wall, ended up with relatively long runtimes and called that a benchmark [1] or maybe not, maybe it's a reduction of actual workload in the system doing computation based on some sort of dynamic number of items so you can't preallocate the intermediate storage and the point's to compare reallocation overhead[2] [2] not very likely though, considering you never need the array in the first place as you're just summing each item with the previous one, then summing every 100 item of that… anyway

rurban 11y ago

If you want to compare fast languages, you need to compare C++ with rust and pony, not swift and go.

pjmlp 11y ago
And include multiple implementations of each one, as that is what really counts.

mangahero 11y ago

How sad. You have to be equivalent for all the languages. Mr. Twitter Developer Advocate at Google

campoy 11y ago
I know, and that is exactly my point. Micro benchmarks across languages with code that doesn't do anything interesting is pretty pointless IMHO. Also, am I a twitter developer advocate now? what is that?
- zaius 11y ago
  I'm guessing s/he's misreading your bio line in your profile.
dang 11y ago
Personal attacks (which that looks like) are not allowed on Hacker News. We detached this subthread from https://news.ycombinator.com/item?id=11098802 https://news.ycombinator.com/item?id=11098802 and marked it off-topic.

oussama-gmd 11y ago

Where is Rust :)

jbeja 11y ago
In academia.
- bluejekyll 11y ago
  This is totally unfair. We're looking at using Rust for production workloads and cross system development.

justinsb 11y ago

As far as micro-benchmarks go, this one is particularly egregious. It is likely completely dominated by the array reallocation. So a runtime that - say - chooses to preallocate an array at size 1000000, or to go to 1000000 at first resize would win at this benchmark. More realistically, a runtime that conserves memory by reallocating arrays with a 1.5x factor vs a 2x factor will appear slower. It's exactly because you don't want to rely on the runtime making the right call for your use-case that you preallocate arrays where it makes a difference (i.e. where you're appending a million items). And 99% of the time you're doing less than 100 items, and so the array reallocation behavior just doesn't matter.

0x434D53 11y ago

So what's the conclusion of this besides some useless numbers?

jbeja 11y ago

What's the point, seriously?

otec 11y ago

Do you actually understand what are you measuring in C++ example ? I dont think you have sufficient knowledge to make any benchmarks.

autoreleasepool 11y ago

FYI, unary operators and C-style for loops have been depreciated in Swift.

jsksma2 11y ago

> "But it adds some checks and GC, so Swift also exhibits a serious slowdown, though well within 2x of C++." Swift does not employ GC, it uses ARC. While that may be a form of GC, it doesn't suffer from the same "slowdown" that GC would in this context because ARC is doing exactly what your base case (C++) does at run-time (ARC is a compile-time tool). I think if you're going to be doing benchmarks, you should have basic knowledge on how each system works first.

chvid 11y ago

I really think this is mostly just testing how well the list/vector/array class is handling being asked to expand itself by one element a million times.