7 ms·

This seems totally misguided and inflammatory. The points made boil down to: 1. Overflow checking adds about a 2x overhead to "+" in a loop 2. Using "reduce"

This seems totally misguided and inflammatory. The points made boil down to: 1. Overflow checking adds about a 2x overhead to "+" in a loop 2. Using "reduce" rather than a loop also has ~2x overhead Neither is really surprising, and hardly seems to warrant the final tangent that "I do not think it is universally a good thing that software should crash when an overflow occurs" since "a mission-critical application could crash because an unimportant function in a secondary routine overflows". What about the mission-critical bugs, especially security bugs, that are caused by overflows? Swift's default behavior, of aggressively checking for overflows, is at least arguable, and there's a school of thought in software engineering that says "don't nail your code to the wall", i.e. don't try to keep it running at all costs when something goes wrong. Let it crash and restart it. Obviously that approach works well in some scenarios and less well in others. The creators of Swift reckon it's a good default for most programmers and I reckon they're onto something. The fact that the overhead is only 2x for a tight inner loop should be a cause for celebration, not dire warnings. And as the author himself shows, it's very easy to disable overflow checking on a per-operation basis when you want to.

planteen 10y ago
> The fact that the overhead is only 2x for a tight inner loop should be a cause for celebration, not dire warnings. And as the author himself shows, it's very easy to disable overflow checking on a per-operation basis when you want to. Agreed. This smells like premature optimization... on a problem that can be solved in O(1) time.
the_af 10y ago
Agreed that it's misguided, inflammatory and presented in the most flame-baity terms. I would have expected this from some random blogger dealing with an unfamiliar programming language, but from a supposed professor of computer science? Does he really expect sympathy when he considers something as trivial as array.reduce(0,+) "obfuscated"? I've never written a single line of Swift in my life and I -- being familiar with FP idioms -- immediately understand what it means. The premise that many abstractions and safety checks come at a cost is reasonable. Unfortunately the rest of his post isn't.
- bluecalm 10y ago
  How is it inflammatory? If you don't like his opinions or writing style then fine but it's in no way inflammatory. He takes a stab at functional programming fashion, that's all there is to it. He also shows significant performance penalty in the default language construct. If anything it's an informative, if a bit opinionated, post.
  - the_af 10y ago
    It's inflammatory because his main point -- that abstractions come at a cost -- is buried in digs at FP, mock surprise, and claiming really simple things are difficult to understand, when they are not (and I know they aren't difficult to him either, so it's also mock ignorance). I don't see any deep insight. The performance loss may very well be caused by an immature Swift compiler, and not ultimately tied to the abstraction, but he mentions nothing of this. I expect this lack of understanding from a novice, not from a CS professor. What he wrote isn't even a "case study", contradicting even the title of his blog.
    - caconym_ 10y ago
      > It's inflammatory because his main point -- that abstractions come at a cost It's also not always correct. See the Rust benchmarks posted here. Someone seeing this article might think it is correct, though, and thusly misinformed, they might avoid "functional-style" programming in the future when in fact many languages manage to provide FP idioms with no performance penalty. It's irresponsible, really. If he wants to state his opinions on FP, that's all well and good, but he should make it clear that they are his opinions and that the performance penalty he saw in the fold case is Swift-specific.
      - nkurz 10y ago
        Given that the title is "Don’t assume that safety comes for free: a Swift case study", I don't think he's being particularly misleading. "Don't assume" usually just means that one should not take something for granted, not that it is never true. And doesn't appending "a Swift case study" makes it fairly clear that he is talking specifically about this case in Swift? Also, he does point out in this paragraph that there is nothing inherently slower about the functional approach: "All these functions achieve nearly the same best performance if you disable safety checks at build time (swift build --configuration release -Xswiftc -Ounchecked). This means, in particular, that we get the fancy abstraction (the reduce approach) for free, as long as we disable safety checks. That makes performance a lot easier to understand." See the Rust benchmarks posted here. Send him a pull request, and I'd bet he'd be happy to include them for comparison and mention them in an addendum: https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/tree/master/2016/12/05 https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/...
TickleSteve 10y ago
Thinking out loud here, but... If you're going to take the overhead of range checking for a value, it may as well be a more appropriate range, not just for overflow conditions... Thinking it would be nice to be able to specify this as part of the type...
- junke 10y ago
  Use Ada.
- nickpsecurity 10y ago
  Exactly. This is what Ada and annotation-based verification do. Usually, there's bad input at one of the interface points (esp function calls or I/O) where some conditional checks can catch it. Not always but most cases.
mikeash 10y ago
I'm puzzled by how many people want their programs to just keep on trucking and try their best after something unexpected occurs. Crashing is not the worst thing your program can do. In fact, I'd argue that it's the second-best thing your program can do, right behind working correctly. Losing data is much worse than crashing, and corrupting data is worse still. Both are strong possibilities once your program's state is something unexpected.
- arcticbull 10y ago
  I agree and disagree. What the author says is true, if a thoroughly irrelevant part of your program (let's say a drawing routine for UI somewhere) hits an overflow and crashes your app in the middle of taking a payment, or in the middle of surgery -- while that much more critical element continued to work perfectly -- that's not ideal. Not all parts of your app make sense to fail-fast. By enforcing this behavior even on peripheral parts of your app, you're taking away the decision making ability about what's important and what isn't from the system designer. I understand that position, though I still think on the whole explicitness is right. Being able to explicitly disable the check when you know what you want is the right balance, whether it's &+ in Swift or (better, imo) .wrapping_add() in Rust.
  - mikeash 10y ago
    Displaying corrupt data in the UI can be just as bad as saving it to disk, if the user actually reads and uses the data you display (and if not, why are you displaying it?). And since the UI almost certainly shares data with the rest of the program, bugs in the UI can corrupt the underlying data. Separation of concerns is great, but if you really want different parts of your system to have different reliability constraints, then you need to truly separate them, for example by running them in different processes. Running them in the same process but having some of the code be more lax about state because it's less important is just begging for horrible bugs.
    - lomnakkus 10y ago
      > Separation of concerns is great, but if you really want different parts of your system to have different reliability constraints, then you need to truly separate them, for example by running them in different processes. Running them in the same process but having some of the code be more lax about state because it's less important is just begging for horrible bugs. Indeed, and you might even want to run them on entirely different and independent systems (connected by some type of network/bus) depending on exactly what bits are critical vs. non-critical.
AndrewStephens 10y ago
I totally agree. The most pernicious defect I ever fixed was a result of a subtle overflow during part of a larger calculation. The erroneous result could cause problems months after the overflow occurred and took me days to track down. The fact that Swift defaults to checking for overflow should be treated as good news.
- planteen 10y ago
  I had a nasty bug like that once as well. It was in some FPGA logic written in VHDL with an overflow on a FIR filter. Recreating the problem was very difficult since it only happened near max signal power with a noisy input. The correct solution there was to saturate, not wrap, which is almost always the case in DSP code.
steveklabnik 10y ago
> Neither is really surprising, It depends! Consider these three functions, in Rust: https://gist.github.com/steveklabnik/da82b51f9dbedd5321924927af091131 https://gist.github.com/steveklabnik/da82b51f9dbedd532192492... One uses a loop, one uses fold with a closure, one uses point-free style. On my machine, summing a vector of a million fives gives running 3 tests test sum_fold ... bench: 147,129 ns/iter (+/- 9,101) test sum_loop ... bench: 148,016 ns/iter (+/- 10,294) test sum_pointfree ... bench: 146,230 ns/iter (+/- 9,114) (there is some variance here between runs, sometimes the +/- for each changes a bit.) Cargo's bench profile doesn't use debug assertions, and hence, no overflow checks by default. Let's turn them on: running 3 tests test sum_fold ... bench: 147,638 ns/iter (+/- 12,654) test sum_loop ... bench: 143,485 ns/iter (+/- 7,255) test sum_pointfree ... bench: 143,050 ns/iter (+/- 7,549) Looks like they're being eliminated by LLVM anyway. Using godbolt, both the loop and the fold versions (I didn't bother to look at pointfree) are both getting vectorized, and while they're not _exactly_ identical, they're quite similar. (disclaimer: YMMV, I might have made a mistake, always inspect assembly for your situation, etc)
- iainmerrick 10y ago
  Good point! You're right, it is a little surprising that reduce() adds any overhead at all in this trivial example. It's interesting that Swift can't seem to fully eliminate that abstraction overhead when runtime checks are enabled -- maybe there's something other than overflow checks that's getting in the way? The author hints that the abstraction cost does vanish when checks are disabled, but doesn't bother showing any numbers.
  - jckarter 10y ago
    Looking at the generated code, we have a couple issues that prevent full optimization. One of them I already knew about (https://bugs.swift.org/browse/SR-2926 https://bugs.swift.org/browse/SR-2926) -- as a debugging aid, we guard every trap we emit with the equivalent of an `asm volatile("")` barrier, because LLVM will otherwise happily fold all the traps together into one trap. We really want every trap to have a distinct address so that the debugger can map that trap back to the source code that emitted it, but the asm barrier prevents other optimizations as well. As for `reduce`, it looks like the compiler does inline away the closure and turn the inner loop into what you would expect, but for some reason there's more pre-matter validation of the array than there is with a for loop. That's a bug; by all means, reduce is intended to optimize the same.
int_19h 10y ago
> Using "reduce" rather than a loop also has ~2x overhead ... Neither is really surprising I disagree. It all depends on where you're coming from, and what your expectations are. In C++, using reduce (std::accumulate) with a lambda is normally as fast as using a loop, because the compiler will inline the lambda into accumulate, and then accumulate into the calling function, producing the same exact loop.
deleted 10y ago
[deleted]
caconym_ 10y ago
It's a little bit surprising that using a fold rather than a simple loop has such a high overhead. I was already pretty sure of the result, but I went ahead and wrote a quick benchmark in Rust, and there is no overhead to using a fold rather than a loop (each iteration is addition of a million values of the stated numeric type): test tests::bench_fold_f64 ... bench: 984,829 ns/iter (+/- 30,827) test tests::bench_fold_i32 ... bench: 84,497 ns/iter (+/- 16,109) test tests::bench_loop_f64 ... bench: 993,406 ns/iter (+/- 45,791) test tests::bench_loop_i32 ... bench: 81,306 ns/iter (+/- 17,815) That this isn't true in Swift is way more interesting to me than the fact that overflow checking has overhead (duh). It's not difficult to come up with some theories as to why this might be the case, but still, if using these "functional" constructs in Swift has such a high overhead, programmers should probably be questioning whether it's worth using them at all. edit: I see that I've been beaten to the punch posting a Rust benchmark :)

planteen 10y ago

> The fact that the overhead is only 2x for a tight inner loop should be a cause for celebration, not dire warnings. And as the author himself shows, it's very easy to disable overflow checking on a per-operation basis when you want to. Agreed. This smells like premature optimization... on a problem that can be solved in O(1) time.

the_af 10y ago

Agreed that it's misguided, inflammatory and presented in the most flame-baity terms. I would have expected this from some random blogger dealing with an unfamiliar programming language, but from a supposed professor of computer science? Does he really expect sympathy when he considers something as trivial as array.reduce(0,+) "obfuscated"? I've never written a single line of Swift in my life and I -- being familiar with FP idioms -- immediately understand what it means. The premise that many abstractions and safety checks come at a cost is reasonable. Unfortunately the rest of his post isn't.

bluecalm 10y ago
How is it inflammatory? If you don't like his opinions or writing style then fine but it's in no way inflammatory. He takes a stab at functional programming fashion, that's all there is to it. He also shows significant performance penalty in the default language construct. If anything it's an informative, if a bit opinionated, post.
- the_af 10y ago
  It's inflammatory because his main point -- that abstractions come at a cost -- is buried in digs at FP, mock surprise, and claiming really simple things are difficult to understand, when they are not (and I know they aren't difficult to him either, so it's also mock ignorance). I don't see any deep insight. The performance loss may very well be caused by an immature Swift compiler, and not ultimately tied to the abstraction, but he mentions nothing of this. I expect this lack of understanding from a novice, not from a CS professor. What he wrote isn't even a "case study", contradicting even the title of his blog.
  - caconym_ 10y ago
    > It's inflammatory because his main point -- that abstractions come at a cost It's also not always correct. See the Rust benchmarks posted here. Someone seeing this article might think it is correct, though, and thusly misinformed, they might avoid "functional-style" programming in the future when in fact many languages manage to provide FP idioms with no performance penalty. It's irresponsible, really. If he wants to state his opinions on FP, that's all well and good, but he should make it clear that they are his opinions and that the performance penalty he saw in the fold case is Swift-specific.
    - nkurz 10y ago
      Given that the title is "Don’t assume that safety comes for free: a Swift case study", I don't think he's being particularly misleading. "Don't assume" usually just means that one should not take something for granted, not that it is never true. And doesn't appending "a Swift case study" makes it fairly clear that he is talking specifically about this case in Swift? Also, he does point out in this paragraph that there is nothing inherently slower about the functional approach: "All these functions achieve nearly the same best performance if you disable safety checks at build time (swift build --configuration release -Xswiftc -Ounchecked). This means, in particular, that we get the fancy abstraction (the reduce approach) for free, as long as we disable safety checks. That makes performance a lot easier to understand." See the Rust benchmarks posted here. Send him a pull request, and I'd bet he'd be happy to include them for comparison and mention them in an addendum: https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/tree/master/2016/12/05 https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/...

TickleSteve 10y ago

Thinking out loud here, but... If you're going to take the overhead of range checking for a value, it may as well be a more appropriate range, not just for overflow conditions... Thinking it would be nice to be able to specify this as part of the type...

junke 10y ago
Use Ada.
nickpsecurity 10y ago
Exactly. This is what Ada and annotation-based verification do. Usually, there's bad input at one of the interface points (esp function calls or I/O) where some conditional checks can catch it. Not always but most cases.

mikeash 10y ago

I'm puzzled by how many people want their programs to just keep on trucking and try their best after something unexpected occurs. Crashing is not the worst thing your program can do. In fact, I'd argue that it's the second-best thing your program can do, right behind working correctly. Losing data is much worse than crashing, and corrupting data is worse still. Both are strong possibilities once your program's state is something unexpected.

arcticbull 10y ago
I agree and disagree. What the author says is true, if a thoroughly irrelevant part of your program (let's say a drawing routine for UI somewhere) hits an overflow and crashes your app in the middle of taking a payment, or in the middle of surgery -- while that much more critical element continued to work perfectly -- that's not ideal. Not all parts of your app make sense to fail-fast. By enforcing this behavior even on peripheral parts of your app, you're taking away the decision making ability about what's important and what isn't from the system designer. I understand that position, though I still think on the whole explicitness is right. Being able to explicitly disable the check when you know what you want is the right balance, whether it's &+ in Swift or (better, imo) .wrapping_add() in Rust.
- mikeash 10y ago
  Displaying corrupt data in the UI can be just as bad as saving it to disk, if the user actually reads and uses the data you display (and if not, why are you displaying it?). And since the UI almost certainly shares data with the rest of the program, bugs in the UI can corrupt the underlying data. Separation of concerns is great, but if you really want different parts of your system to have different reliability constraints, then you need to truly separate them, for example by running them in different processes. Running them in the same process but having some of the code be more lax about state because it's less important is just begging for horrible bugs.
  - lomnakkus 10y ago
    > Separation of concerns is great, but if you really want different parts of your system to have different reliability constraints, then you need to truly separate them, for example by running them in different processes. Running them in the same process but having some of the code be more lax about state because it's less important is just begging for horrible bugs. Indeed, and you might even want to run them on entirely different and independent systems (connected by some type of network/bus) depending on exactly what bits are critical vs. non-critical.

AndrewStephens 10y ago

I totally agree. The most pernicious defect I ever fixed was a result of a subtle overflow during part of a larger calculation. The erroneous result could cause problems months after the overflow occurred and took me days to track down. The fact that Swift defaults to checking for overflow should be treated as good news.

planteen 10y ago
I had a nasty bug like that once as well. It was in some FPGA logic written in VHDL with an overflow on a FIR filter. Recreating the problem was very difficult since it only happened near max signal power with a noisy input. The correct solution there was to saturate, not wrap, which is almost always the case in DSP code.

steveklabnik 10y ago

> Neither is really surprising, It depends! Consider these three functions, in Rust: https://gist.github.com/steveklabnik/da82b51f9dbedd5321924927af091131 https://gist.github.com/steveklabnik/da82b51f9dbedd532192492... One uses a loop, one uses fold with a closure, one uses point-free style. On my machine, summing a vector of a million fives gives running 3 tests test sum_fold ... bench: 147,129 ns/iter (+/- 9,101) test sum_loop ... bench: 148,016 ns/iter (+/- 10,294) test sum_pointfree ... bench: 146,230 ns/iter (+/- 9,114) (there is some variance here between runs, sometimes the +/- for each changes a bit.) Cargo's bench profile doesn't use debug assertions, and hence, no overflow checks by default. Let's turn them on: running 3 tests test sum_fold ... bench: 147,638 ns/iter (+/- 12,654) test sum_loop ... bench: 143,485 ns/iter (+/- 7,255) test sum_pointfree ... bench: 143,050 ns/iter (+/- 7,549) Looks like they're being eliminated by LLVM anyway. Using godbolt, both the loop and the fold versions (I didn't bother to look at pointfree) are both getting vectorized, and while they're not _exactly_ identical, they're quite similar. (disclaimer: YMMV, I might have made a mistake, always inspect assembly for your situation, etc)

iainmerrick 10y ago
Good point! You're right, it is a little surprising that reduce() adds any overhead at all in this trivial example. It's interesting that Swift can't seem to fully eliminate that abstraction overhead when runtime checks are enabled -- maybe there's something other than overflow checks that's getting in the way? The author hints that the abstraction cost does vanish when checks are disabled, but doesn't bother showing any numbers.
- jckarter 10y ago
  Looking at the generated code, we have a couple issues that prevent full optimization. One of them I already knew about (https://bugs.swift.org/browse/SR-2926 https://bugs.swift.org/browse/SR-2926) -- as a debugging aid, we guard every trap we emit with the equivalent of an `asm volatile("")` barrier, because LLVM will otherwise happily fold all the traps together into one trap. We really want every trap to have a distinct address so that the debugger can map that trap back to the source code that emitted it, but the asm barrier prevents other optimizations as well. As for `reduce`, it looks like the compiler does inline away the closure and turn the inner loop into what you would expect, but for some reason there's more pre-matter validation of the array than there is with a for loop. That's a bug; by all means, reduce is intended to optimize the same.

int_19h 10y ago

> Using "reduce" rather than a loop also has ~2x overhead ... Neither is really surprising I disagree. It all depends on where you're coming from, and what your expectations are. In C++, using reduce (std::accumulate) with a lambda is normally as fast as using a loop, because the compiler will inline the lambda into accumulate, and then accumulate into the calling function, producing the same exact loop.

deleted 10y ago

[deleted]

caconym_ 10y ago

It's a little bit surprising that using a fold rather than a simple loop has such a high overhead. I was already pretty sure of the result, but I went ahead and wrote a quick benchmark in Rust, and there is no overhead to using a fold rather than a loop (each iteration is addition of a million values of the stated numeric type): test tests::bench_fold_f64 ... bench: 984,829 ns/iter (+/- 30,827) test tests::bench_fold_i32 ... bench: 84,497 ns/iter (+/- 16,109) test tests::bench_loop_f64 ... bench: 993,406 ns/iter (+/- 45,791) test tests::bench_loop_i32 ... bench: 81,306 ns/iter (+/- 17,815) That this isn't true in Swift is way more interesting to me than the fact that overflow checking has overhead (duh). It's not difficult to come up with some theories as to why this might be the case, but still, if using these "functional" constructs in Swift has such a high overhead, programmers should probably be questioning whether it's worth using them at all. edit: I see that I've been beaten to the punch posting a Rust benchmark :)