12 ms·

Implementing "defer"

48 points by lerno 6y ago

pansa2 6y ago
> Go has it's own special defer which only fires on function end, otherwise defer has consistent "execute at scope end" semantics. Does anyone know why Go’s `defer` semantics is so unintuitive? Why is `defer` function-based rather than block-based, in a language where variables are block-scoped?
- lerno 6y ago
  If I may venture a guess, it is because this way there is no limit in regard to jumps within or out of a function. Since Go retains `goto`, function based is easier to implement. In addition, non-local jumps, such ones constructed by setjmp, are fairly easily easy to resolve (just say that the jump will release all queued defers before the jump). But I've also heard the opinion that some find the function based `defer` easier to reason about, so maybe that weighs in as well.
  - iainmerrick 6y ago
    I suspect that it’s a side-effect of Go’s origin in the Plan 9 compiler chain. This is really just speculation, but I’ll try to justify it. First, early versions of Go strictly required a “return” as the final statement in a function, rather than just checking that each branch terminated successfully (i.e. you couldn’t return from both branches of an if/else, you also needed a redundant return statement afterwards). So I think initially the compiler didn’t perform the kind of escape analysis that would make block-scoped “defer” easy to implement. Second, block-scoped defer is really equivalent to RAII via constructors and destructors in C++. Go was specifically designed as an anti-C++, an attempt to return to the roots of C and improve on it, so the designers would have been very wary of copying C++ features like RAII. I think it was a mis-step, and they should have gone with block-scoped. Other languages have copied Go’s syntax, but they’ve uniformly opted for C++’s semantics, which in this instance are much easier to reason about (definitely not always the case with C++!)
    - knorker 6y ago
      > Go was specifically designed as an anti-C++ Ha! And still they managed to reimplement RAII and exceptions, only very very poorly.
- eknkc 6y ago
  One benefit of function based defer is that you can use anonymous functions to create hacky block-based defers. If the language only has block based defers then that's it. You have that and only that. What if I create a resource in an if block and want to defer closing it when the function ends?
  - dthul 6y ago
    Depending on how the defer implementation looks like the same could be achieved with block-level defer by having the defer's "owner" (e.g. some object with RAII semantics) in the outer scope.
  - bbatha 6y ago
    You can get function based defers pretty easily with block ones too func Deferred() { defers := []func(){} defer func() { for _, def := range defers { def() } } for { cleanup, done := work() defers = append(defers, cleanup) if done { break } } } Edit: this is how defers were implemented in the compiler until a few versions ago and will fall back to this if you defer in a loop
- tonyedgecombe 6y ago
  I assumed it was performance, my understanding is it keeps a stack of statements to execute a function end. Perhaps the cost was too high to do that at the end of each block.
  - unscaled 6y ago
    This is not about (runtime) performance. Like the the article mentions, the block-end version lets you implement everything without dynamic allocations and (I guess) pointer de-reference. The C3 implementation of block-end defer described in the article is zero-cost, as far as I can see. I suspect the real reason is the driving force behind a great deal of the early Go 1.x design choices: making the compiler simpler. Or, if you wish: _compiler_ performance. Go 2.x is shaping up to be a different beast, but Go 1.x had an extremely strong design philosophy favoring compiler simplicity (which often equates to compilation speed) over nearly every other concern, including runtime performance. That's exactly one of the main argument that was used against having generics: (non-erased) generics significantly improve performance, but slow down the compiler. I'm personally very much in disagreement with this philosophy, but I don't think all these design choices are baffling or even misguided - they clearly achieve what they were set to do.
    - sltkr 6y ago
      I think the reason is much simpler: block-scoped defer isn't useful if you statements appear in a nested construct. Example: fd := OpenSocket(...); if fd != 0 { defer CloseSocket(fd); } // do stuff For the same reason, Go's defer cannot be implemented without dynamic allocations. For example, how would you implement: func foo(int n) void { for i := 0; i < n; i++ { var res = bar(n); defer fmt.Println(res); } } This calls bar(0), bar(1), etc. then prints the results of bar(n - 1), bar(n - 2), etc. There is no way to implement this without some way to store the intermediate values.
      - ttt0 6y ago
        Compiler should be able to figure out at compile time whether the dynamic allocation is needed. But maybe the Go compiler already does this, I don't know.
- orisho 6y ago
  defer also is used to recover from panics, which makes much more sense in a function end context. For example, when writing code that performs an operation with the potential to panic, but that you want to convert to an error, something you can do is name the error return value and then modify it in the defer, after recovering from a panic, according to the cause of the panic. It's not obvious that that pattern would work well if defer could also execute on scope end - what does recover do then? Will it affect the following recovers? Also, if the function continues executing but you modified some variables, it makes it complicated to reason about what could be in those variables. Defer running at function end is much more easier to reason about: functions execute their standard control flow until a return or panic is hit, and then defers are executed, without returning to standard control flow.
knorker 6y ago
The fact that Go's defer runs at function end instead of scope end is, in my opinion having coded and reviewed huge amounts of Go code, a bug. It's almost never what you want, and it requires terrible gymnastics whenever the difference matters. So in C++ implementing defer the right way is to just register a std::function that gets run from the class destructor. I've used it as an alternative to wrapping a resource, or timing a function call. Only gotcha is to be careful about throwing from within the destructor. Or to put another way: don't.
- jerf 6y ago
  I'd like "defer_func" to be an option; I do have some code that seriously uses it that way. But all in all defer defaulting to scope would be a lot more useful. I've written some functions-in-functions solely to get "scope-like" deferral. Now, so far, none of them have ever survived refactoring, in that they've always been lifted up to become their own top-level functions where defer resumes working like I wanted anyhow. But I've also got a ton of code where the difference doesn't much matter (a function with only one scope, or where defer is only on the outer-most scope anyhow), and it still seems like scope would be a better default.
  - knorker 6y ago
    I've written this a few times: class defer{public:defer(std::function<()> f):f_(std::move(f)){} ~defer(){f_();}private:std::function f_;}; Then fd = socket(…); defer _([]{close(fd);}); But yeah, most of the time a refactor will instead create a proper wrapper class. It's rare that I want to RAII something, but only in some cases. E.g. I probably want RAII on FDs always, with proper ownership, not merely in some cases. And this is also what Go missed. I'm a big fan of RAII.
    - gpderetta 6y ago
      No need to box with std::function. Since C++17 this is enough: template<class Fn> struct defer { Fn fn; ~defer() { fn(); } }; template<class Fn> defer(Fn) -> defer<Fn>; int main() { defer _ { [] { std::cout << "exit"; }}; } Although you might want to add the ability to dismiss() the guard. edit: a more complete implementation: https://godbolt.org/z/6675K1 https://godbolt.org/z/6675K1 . I like to name my guards. Making them anonymous is left as an exercise for the reader.
gw 6y ago
Nim's defer just wraps the current scope in a try/finally, with the deferred code running in the finally. It is probably better just to use try/finally directly because it's more explicit about what is in the try block. It's not worth it to obscure that just to avoid a new level of indentation...
- harikb 6y ago
  Came here to note this exact difference with respect to finally Consider this code fp = os.open(x) // imagine file open defer fp.close() fp.read() With defer, one would think I can simply wrap this in a for-loop if I want to open and read a bunch of files. Go doesn’t promise this, but not clear until linter complains. In languages were it “ends at scope”, this is still wrong. If we wrote it as finally, dev would know finally is outside the loop or they need to wrap in another sub-scope inside the loop {}
  - iainmerrick 6y ago
    In languages were it “ends at scope”, this is still wrong. I don’t quite follow -- what would go wrong? If you put that in a loop body, won’t you get one open() and one close() per iteration, as intended?
    - deleted 6y ago
      [deleted]
    - gw 6y ago
      Not the commenter but which one of these is the defer generating? // option 1 try { fp = os.open(x) fp.read() } finally { fp.close() } // option 2 fp = os.open(x) try { fp.read() } finally { fp.close() } Should we close if open fails? Maybe, maybe not, but with try/finally it is obvious which one it is doing.
      - CodesInChaos 6y ago
        The first one doesn't compile in any language that requires variables to be initialized before use.
ourcat 6y ago
Might be useful to mention '.. in Go' in the title. I clicked on it with the assumption it was about loading resources in `script` tags with the defer or async attributes.
- omni 6y ago
  Might be useful to read the article before suggesting a headline for it. The article's about implementing defer in C2.
  - ourcat 6y ago
    My point was about the wording in the title here, before I clicked on to the article. It was by then reading said article when I saw it was about Go and not JS loading. :)
Jasper_ 6y ago
I don't know if you need to handle break/continue specially during the AST parse. Once you do flow analysis and have your basic blocks, it should be pretty easy to place your lexical defers into the right spots, on the graph edges.
- Nuoji 6y ago
  It assumes flow analysis is done though.
stevemk14ebr 6y ago
Here my C++ defer implementation https://twitter.com/stevemk14ebr/status/1347408381301190658?s=19 https://twitter.com/stevemk14ebr/status/1347408381301190658?...
- unscaled 6y ago
  Boost has had the BOOST_SCOPE_EXIT macro at least since Boost 1.38 (February 2009), a few months before the first public announcement of Go. Judging by the copyright, the earliest versions of this library (before acceptance into Boost) are from 2006, before Pike et. al. even started working on Go internally in Google. BOOST_SCOPE_EXIT in turn is based on Andrei Alexandrescu's pre-closure ScopeGuard implementation from 2000. Granted, both BOOST_SCOPE_EXIT and your defer macro work at block end, not function end, but I agree with OP that this is practically always the better approach, so I don't think we should look at this thing as C++ trying to copy go features with lousy hacks. It's probably the other way around, if anything. And modern implentations in C++ (like yours) are extremely straightforward.
- jerf 6y ago
  The problem isn't that C++ can't do "defer"... the problem is that C++ can do everything, and all those "everythings" are constantly interacting because code bases will use all that "everything" fairly routinely. Just glancing at a particular defer implementation won't tell you how it interacts with everything else, let alone how it will interact with whatever strange things your own codebase has. Go's defer function is very predictable, not so much because Go's defer is magic but precisely because it isn't. There aren't a ton of features for it to interact with. This also isn't a special virtue of Go; IMHO it's a special (not unique, but unusual) failing of C++. It was insanely complicated already in the 90s and every standard since then has only made it worse.
- goalieca 6y ago
  std::shared_ptr<void> foo(nullptr, [&bar] (void*) { printf(":) %s\n", bar); }); the downside is having to write nullptr instead of just the closure. Something like this really should be part of the standard.
- sltkr 6y ago
jstimpfle 6y ago
Slightly OT: I don't like defer much, it's just another way to do automated RAII style cleanups. It has slightly different tradeoffs than RAII (it's more of an "in-line" way of coding things) but in the end it's another high-level feature that obfuscates the actual control flow. To be clear, I don't think that chains of cleanup-labels with corresponding gotos are better. As a long-time OOP basher, I find myself using more and more explicit state structures with "object" semantics: There's a destructor that cleans the state up from every possible configuration except "uninitialized". As far as possible, objects should already be considered initialized when cleared to 0. (This idiom is called ZII, or zero-is-initialization). A problem with defer is that if one has to split what the function does in multiple functions, defer breaks, and one needs a completely different solution (explicit state structure). Not to praise it too much but with RAII you don't pay that cost. In other words, as so many language features, defer offers to save early typing work early, at the expense of later friction, or even of requiring a re-write, when the times comes to solidify the code structure. Disclaimer: I've never actually typed the word defer in any programming language. :-)
bmn__ 6y ago
Achieve maximum overkill with the generalisation of this concept in Raku. "defer" looks like a child's toy in comparison. https://docs.raku.org/language/phasers https://docs.raku.org/language/phasers
- lizmat 6y ago
  Indeed! :-) FWIW, the "defer" functionality is implemented as the LEAVE phaser in Raku: https://docs.raku.org/language/phasers#index-entry-Phasers__LEAVE-LEAVE https://docs.raku.org/language/phasers#index-entry-Phasers__...

48 points by lerno 6y ago

> Go has it's own special defer which only fires on function end, otherwise defer has consistent "execute at scope end" semantics. Does anyone know why Go’s `defer` semantics is so unintuitive? Why is `defer` function-based rather than block-based, in a language where variables are block-scoped?

lerno 6y ago
If I may venture a guess, it is because this way there is no limit in regard to jumps within or out of a function. Since Go retains `goto`, function based is easier to implement. In addition, non-local jumps, such ones constructed by setjmp, are fairly easily easy to resolve (just say that the jump will release all queued defers before the jump). But I've also heard the opinion that some find the function based `defer` easier to reason about, so maybe that weighs in as well.
- iainmerrick 6y ago
  I suspect that it’s a side-effect of Go’s origin in the Plan 9 compiler chain. This is really just speculation, but I’ll try to justify it. First, early versions of Go strictly required a “return” as the final statement in a function, rather than just checking that each branch terminated successfully (i.e. you couldn’t return from both branches of an if/else, you also needed a redundant return statement afterwards). So I think initially the compiler didn’t perform the kind of escape analysis that would make block-scoped “defer” easy to implement. Second, block-scoped defer is really equivalent to RAII via constructors and destructors in C++. Go was specifically designed as an anti-C++, an attempt to return to the roots of C and improve on it, so the designers would have been very wary of copying C++ features like RAII. I think it was a mis-step, and they should have gone with block-scoped. Other languages have copied Go’s syntax, but they’ve uniformly opted for C++’s semantics, which in this instance are much easier to reason about (definitely not always the case with C++!)
  - knorker 6y ago
    > Go was specifically designed as an anti-C++ Ha! And still they managed to reimplement RAII and exceptions, only very very poorly.
eknkc 6y ago
One benefit of function based defer is that you can use anonymous functions to create hacky block-based defers. If the language only has block based defers then that's it. You have that and only that. What if I create a resource in an if block and want to defer closing it when the function ends?
- dthul 6y ago
  Depending on how the defer implementation looks like the same could be achieved with block-level defer by having the defer's "owner" (e.g. some object with RAII semantics) in the outer scope.
- bbatha 6y ago
  You can get function based defers pretty easily with block ones too func Deferred() { defers := []func(){} defer func() { for _, def := range defers { def() } } for { cleanup, done := work() defers = append(defers, cleanup) if done { break } } } Edit: this is how defers were implemented in the compiler until a few versions ago and will fall back to this if you defer in a loop
tonyedgecombe 6y ago
I assumed it was performance, my understanding is it keeps a stack of statements to execute a function end. Perhaps the cost was too high to do that at the end of each block.
- unscaled 6y ago
  This is not about (runtime) performance. Like the the article mentions, the block-end version lets you implement everything without dynamic allocations and (I guess) pointer de-reference. The C3 implementation of block-end defer described in the article is zero-cost, as far as I can see. I suspect the real reason is the driving force behind a great deal of the early Go 1.x design choices: making the compiler simpler. Or, if you wish: _compiler_ performance. Go 2.x is shaping up to be a different beast, but Go 1.x had an extremely strong design philosophy favoring compiler simplicity (which often equates to compilation speed) over nearly every other concern, including runtime performance. That's exactly one of the main argument that was used against having generics: (non-erased) generics significantly improve performance, but slow down the compiler. I'm personally very much in disagreement with this philosophy, but I don't think all these design choices are baffling or even misguided - they clearly achieve what they were set to do.
  - sltkr 6y ago
    I think the reason is much simpler: block-scoped defer isn't useful if you statements appear in a nested construct. Example: fd := OpenSocket(...); if fd != 0 { defer CloseSocket(fd); } // do stuff For the same reason, Go's defer cannot be implemented without dynamic allocations. For example, how would you implement: func foo(int n) void { for i := 0; i < n; i++ { var res = bar(n); defer fmt.Println(res); } } This calls bar(0), bar(1), etc. then prints the results of bar(n - 1), bar(n - 2), etc. There is no way to implement this without some way to store the intermediate values.
    - ttt0 6y ago
      Compiler should be able to figure out at compile time whether the dynamic allocation is needed. But maybe the Go compiler already does this, I don't know.
orisho 6y ago
defer also is used to recover from panics, which makes much more sense in a function end context. For example, when writing code that performs an operation with the potential to panic, but that you want to convert to an error, something you can do is name the error return value and then modify it in the defer, after recovering from a panic, according to the cause of the panic. It's not obvious that that pattern would work well if defer could also execute on scope end - what does recover do then? Will it affect the following recovers? Also, if the function continues executing but you modified some variables, it makes it complicated to reason about what could be in those variables. Defer running at function end is much more easier to reason about: functions execute their standard control flow until a return or panic is hit, and then defers are executed, without returning to standard control flow.

knorker 6y ago

The fact that Go's defer runs at function end instead of scope end is, in my opinion having coded and reviewed huge amounts of Go code, a bug. It's almost never what you want, and it requires terrible gymnastics whenever the difference matters. So in C++ implementing defer the right way is to just register a std::function that gets run from the class destructor. I've used it as an alternative to wrapping a resource, or timing a function call. Only gotcha is to be careful about throwing from within the destructor. Or to put another way: don't.

jerf 6y ago
I'd like "defer_func" to be an option; I do have some code that seriously uses it that way. But all in all defer defaulting to scope would be a lot more useful. I've written some functions-in-functions solely to get "scope-like" deferral. Now, so far, none of them have ever survived refactoring, in that they've always been lifted up to become their own top-level functions where defer resumes working like I wanted anyhow. But I've also got a ton of code where the difference doesn't much matter (a function with only one scope, or where defer is only on the outer-most scope anyhow), and it still seems like scope would be a better default.
- knorker 6y ago
  I've written this a few times: class defer{public:defer(std::function<()> f):f_(std::move(f)){} ~defer(){f_();}private:std::function f_;}; Then fd = socket(…); defer _([]{close(fd);}); But yeah, most of the time a refactor will instead create a proper wrapper class. It's rare that I want to RAII something, but only in some cases. E.g. I probably want RAII on FDs always, with proper ownership, not merely in some cases. And this is also what Go missed. I'm a big fan of RAII.
  - gpderetta 6y ago
    No need to box with std::function. Since C++17 this is enough: template<class Fn> struct defer { Fn fn; ~defer() { fn(); } }; template<class Fn> defer(Fn) -> defer<Fn>; int main() { defer _ { [] { std::cout << "exit"; }}; } Although you might want to add the ability to dismiss() the guard. edit: a more complete implementation: https://godbolt.org/z/6675K1 https://godbolt.org/z/6675K1 . I like to name my guards. Making them anonymous is left as an exercise for the reader.

gw 6y ago

Nim's defer just wraps the current scope in a try/finally, with the deferred code running in the finally. It is probably better just to use try/finally directly because it's more explicit about what is in the try block. It's not worth it to obscure that just to avoid a new level of indentation...

harikb 6y ago
Came here to note this exact difference with respect to finally Consider this code fp = os.open(x) // imagine file open defer fp.close() fp.read() With defer, one would think I can simply wrap this in a for-loop if I want to open and read a bunch of files. Go doesn’t promise this, but not clear until linter complains. In languages were it “ends at scope”, this is still wrong. If we wrote it as finally, dev would know finally is outside the loop or they need to wrap in another sub-scope inside the loop {}
- iainmerrick 6y ago
  In languages were it “ends at scope”, this is still wrong. I don’t quite follow -- what would go wrong? If you put that in a loop body, won’t you get one open() and one close() per iteration, as intended?
  - deleted 6y ago
    [deleted]
  - gw 6y ago
    Not the commenter but which one of these is the defer generating? // option 1 try { fp = os.open(x) fp.read() } finally { fp.close() } // option 2 fp = os.open(x) try { fp.read() } finally { fp.close() } Should we close if open fails? Maybe, maybe not, but with try/finally it is obvious which one it is doing.
    - CodesInChaos 6y ago
      The first one doesn't compile in any language that requires variables to be initialized before use.

ourcat 6y ago

Might be useful to mention '.. in Go' in the title. I clicked on it with the assumption it was about loading resources in `script` tags with the defer or async attributes.

omni 6y ago
Might be useful to read the article before suggesting a headline for it. The article's about implementing defer in C2.
- ourcat 6y ago
  My point was about the wording in the title here, before I clicked on to the article. It was by then reading said article when I saw it was about Go and not JS loading. :)

Jasper_ 6y ago

I don't know if you need to handle break/continue specially during the AST parse. Once you do flow analysis and have your basic blocks, it should be pretty easy to place your lexical defers into the right spots, on the graph edges.

Nuoji 6y ago
It assumes flow analysis is done though.

stevemk14ebr 6y ago

Here my C++ defer implementation https://twitter.com/stevemk14ebr/status/1347408381301190658?s=19 https://twitter.com/stevemk14ebr/status/1347408381301190658?...

unscaled 6y ago
Boost has had the BOOST_SCOPE_EXIT macro at least since Boost 1.38 (February 2009), a few months before the first public announcement of Go. Judging by the copyright, the earliest versions of this library (before acceptance into Boost) are from 2006, before Pike et. al. even started working on Go internally in Google. BOOST_SCOPE_EXIT in turn is based on Andrei Alexandrescu's pre-closure ScopeGuard implementation from 2000. Granted, both BOOST_SCOPE_EXIT and your defer macro work at block end, not function end, but I agree with OP that this is practically always the better approach, so I don't think we should look at this thing as C++ trying to copy go features with lousy hacks. It's probably the other way around, if anything. And modern implentations in C++ (like yours) are extremely straightforward.
jerf 6y ago
The problem isn't that C++ can't do "defer"... the problem is that C++ can do everything, and all those "everythings" are constantly interacting because code bases will use all that "everything" fairly routinely. Just glancing at a particular defer implementation won't tell you how it interacts with everything else, let alone how it will interact with whatever strange things your own codebase has. Go's defer function is very predictable, not so much because Go's defer is magic but precisely because it isn't. There aren't a ton of features for it to interact with. This also isn't a special virtue of Go; IMHO it's a special (not unique, but unusual) failing of C++. It was insanely complicated already in the 90s and every standard since then has only made it worse.
goalieca 6y ago
std::shared_ptr<void> foo(nullptr, [&bar] (void*) { printf(":) %s\n", bar); }); the downside is having to write nullptr instead of just the closure. Something like this really should be part of the standard.
sltkr 6y ago

jstimpfle 6y ago

Slightly OT: I don't like defer much, it's just another way to do automated RAII style cleanups. It has slightly different tradeoffs than RAII (it's more of an "in-line" way of coding things) but in the end it's another high-level feature that obfuscates the actual control flow. To be clear, I don't think that chains of cleanup-labels with corresponding gotos are better. As a long-time OOP basher, I find myself using more and more explicit state structures with "object" semantics: There's a destructor that cleans the state up from every possible configuration except "uninitialized". As far as possible, objects should already be considered initialized when cleared to 0. (This idiom is called ZII, or zero-is-initialization). A problem with defer is that if one has to split what the function does in multiple functions, defer breaks, and one needs a completely different solution (explicit state structure). Not to praise it too much but with RAII you don't pay that cost. In other words, as so many language features, defer offers to save early typing work early, at the expense of later friction, or even of requiring a re-write, when the times comes to solidify the code structure. Disclaimer: I've never actually typed the word defer in any programming language. :-)

bmn__ 6y ago

Achieve maximum overkill with the generalisation of this concept in Raku. "defer" looks like a child's toy in comparison. https://docs.raku.org/language/phasers https://docs.raku.org/language/phasers

lizmat 6y ago
Indeed! :-) FWIW, the "defer" functionality is implemented as the LEAVE phaser in Raku: https://docs.raku.org/language/phasers#index-entry-Phasers__LEAVE-LEAVE https://docs.raku.org/language/phasers#index-entry-Phasers__...