4 ms·

Curious if anyone knows why the swift version is slow? Is it the casting from subsequence to a string?

bhauer 4y ago
As I wrote elsewhere in this thread (though I was downvoted), I suspect the reason is that this benchmark is measuring the total execution time of a single run of an executable. This would include any time spent by the executable to bootstrap its runtime environment, initiate a virtual machine, and whatever else it needs to do in addition to the relevant code at hand to process the string. Such a test will favor implementations that have extremely svelte boostrapping, allowing them to immediately begin executing the relevant code and return a result. I feel a more useful test would be for the relevant string processing code to be run tens to thousands of times within the process itself, so as to diminish the relative importance of boostrap/warmup code and/or runtime optimization performed by VMs. Unless, of course, part of the intent was specifically to measure the impact of the bootstrapping.
- kmonsen 4y ago
  I was going to make a general comment about this too. It’s a huge penalty for Java too and has no correlation for how well it performs outside of toy benchmarks
  - jiggawatts 4y ago
    … unless you’re using Java in the same way, such as an Azure Function or AWS Lambda. Or simply as a command-line tool. IMHO, VM startup time should be included and the first few passes (before JIT kicks in) should also be included. Java has supposedly “nearly C-like performance” until you read the fine print. (This should apply to C# as well.)
  - igouy 4y ago
    > a huge penalty Is that just your assumption or have you measured that penalty for this tiny tiny program? Might "the JVM's slow start time" in this case be insignificant? https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#jvm-startup-time https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- josephg 4y ago
  I agree that the benchmark should be run for much longer to get a useful result. But why does Swift have a long startup time in the first place? Shouldn’t it start near instantly like the C and Rust programs?
  - ksbrooksjr 4y ago
    I don't think Swift's performance is due to start up time at all. I actually cloned the repo, and ran the benchmark and found that Swift's execution time scales drastically with the size of the input. The Swift team actually boasts about its quick start up time on the official website [1]. I ran a simple hello world benchmark to gauge Swift's start up time and got an output of 13 milliseconds. echo 'print("hello world")' > hello.swift && swiftc hello.swift -O -o hello time ./hello [1] https://www.swift.org/server/ https://www.swift.org/server/
    - saagarjha 4y ago
      Yeah the startup time for the Swift code is on the order of several milliseconds on my computer, the profiler running at 1000 Hz only gets four samples off of it
- BiteCode_dev 4y ago
  In that case, it would change the Python result dramatically as well. But I don't think it would be fair: I use Python a lot, and if it's for scripting, you are happy with the fact it's very easy to write. However, it's slow to start, and you pay that each time you run the script. To me, it makes sense in this exercice, which is heavily leaning toward scripting, we see the price of the VM start in the overall profiling. Otherwise, let's use pypy, warm it up, a few 1000 times, and you may get closer to Go times. But we don't use pypy for scripting.
saagarjha 4y ago
Here's a very rough breakdown of the operations it does: 1.33 s 56.7% specialized Collection<>.split(separator:maxSplits:omittingEmptySubsequences:) 263.00 ms 11.2% Substring.lowercased() 226.00 ms 9.6% specialized Dictionary.subscript.modify 189.00 ms 8.0% readLine(strippingNewline:) As you can see, calling split on the string is really slow. The reason it is slow is that it's using the generic implementation from Collection, rather than String, which doesn't really know anything about how String works. So to do the split it's doing a linear march down the string calling formIndex(after:), then subscripting to see if there's a space character there using Unicode-aware string comparison on that one Character. Swift is really nice that it gives you "default" implementations of things for free if you conform to the right things in the protocol hierarchy. But, and this is kind of unfortunately a common bottleneck, if you "know" more you should really specialize the implementation to use a more optimized path that can use all the context that is available. For example, in this case a "split" implementation should really do its own substring matching and splitting rather than having Collection call it through its slow sequential indexing API. Probably something for the stdlib to look at, I guess. The rest of the things are also not too unfamiliar but I'll go over them one by one; lowercased() is slow because it creates a new Substring and then a new String, plus Unicode stuff. Dictionary accesses are slow because Swift uses a secure but not very performant hash function. readLine is slow because the lines are short and the call to getline is not particularly optimized on macOS, reallocs and locks internally, then the result is used to create a new String with the newline sliced off.
- ppeetteerr 4y ago
  Thank you for this response! It's sad to see that Swift is so slow in this particular case given that it has the potential to be such an optimized language (strong typing, compilation, backing from Apple).

As I wrote elsewhere in this thread (though I was downvoted), I suspect the reason is that this benchmark is measuring the total execution time of a single run of an executable. This would include any time spent by the executable to bootstrap its runtime environment, initiate a virtual machine, and whatever else it needs to do in addition to the relevant code at hand to process the string. Such a test will favor implementations that have extremely svelte boostrapping, allowing them to immediately begin executing the relevant code and return a result. I feel a more useful test would be for the relevant string processing code to be run tens to thousands of times within the process itself, so as to diminish the relative importance of boostrap/warmup code and/or runtime optimization performed by VMs. Unless, of course, part of the intent was specifically to measure the impact of the bootstrapping.

kmonsen 4y ago
I was going to make a general comment about this too. It’s a huge penalty for Java too and has no correlation for how well it performs outside of toy benchmarks
- jiggawatts 4y ago
  … unless you’re using Java in the same way, such as an Azure Function or AWS Lambda. Or simply as a command-line tool. IMHO, VM startup time should be included and the first few passes (before JIT kicks in) should also be included. Java has supposedly “nearly C-like performance” until you read the fine print. (This should apply to C# as well.)
- igouy 4y ago
  > a huge penalty Is that just your assumption or have you measured that penalty for this tiny tiny program? Might "the JVM's slow start time" in this case be insignificant? https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#jvm-startup-time https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
josephg 4y ago
I agree that the benchmark should be run for much longer to get a useful result. But why does Swift have a long startup time in the first place? Shouldn’t it start near instantly like the C and Rust programs?
- ksbrooksjr 4y ago
  I don't think Swift's performance is due to start up time at all. I actually cloned the repo, and ran the benchmark and found that Swift's execution time scales drastically with the size of the input. The Swift team actually boasts about its quick start up time on the official website [1]. I ran a simple hello world benchmark to gauge Swift's start up time and got an output of 13 milliseconds. echo 'print("hello world")' > hello.swift && swiftc hello.swift -O -o hello time ./hello [1] https://www.swift.org/server/ https://www.swift.org/server/
  - saagarjha 4y ago
    Yeah the startup time for the Swift code is on the order of several milliseconds on my computer, the profiler running at 1000 Hz only gets four samples off of it
BiteCode_dev 4y ago
In that case, it would change the Python result dramatically as well. But I don't think it would be fair: I use Python a lot, and if it's for scripting, you are happy with the fact it's very easy to write. However, it's slow to start, and you pay that each time you run the script. To me, it makes sense in this exercice, which is heavily leaning toward scripting, we see the price of the VM start in the overall profiling. Otherwise, let's use pypy, warm it up, a few 1000 times, and you may get closer to Go times. But we don't use pypy for scripting.

saagarjha 4y ago

Here's a very rough breakdown of the operations it does: 1.33 s 56.7% specialized Collection<>.split(separator:maxSplits:omittingEmptySubsequences:) 263.00 ms 11.2% Substring.lowercased() 226.00 ms 9.6% specialized Dictionary.subscript.modify 189.00 ms 8.0% readLine(strippingNewline:) As you can see, calling split on the string is really slow. The reason it is slow is that it's using the generic implementation from Collection, rather than String, which doesn't really know anything about how String works. So to do the split it's doing a linear march down the string calling formIndex(after:), then subscripting to see if there's a space character there using Unicode-aware string comparison on that one Character. Swift is really nice that it gives you "default" implementations of things for free if you conform to the right things in the protocol hierarchy. But, and this is kind of unfortunately a common bottleneck, if you "know" more you should really specialize the implementation to use a more optimized path that can use all the context that is available. For example, in this case a "split" implementation should really do its own substring matching and splitting rather than having Collection call it through its slow sequential indexing API. Probably something for the stdlib to look at, I guess. The rest of the things are also not too unfamiliar but I'll go over them one by one; lowercased() is slow because it creates a new Substring and then a new String, plus Unicode stuff. Dictionary accesses are slow because Swift uses a secure but not very performant hash function. readLine is slow because the lines are short and the call to getline is not particularly optimized on macOS, reallocs and locks internally, then the result is used to create a new String with the newline sliced off.

ppeetteerr 4y ago
Thank you for this response! It's sad to see that Swift is so slow in this particular case given that it has the potential to be such an optimized language (strong typing, compilation, backing from Apple).