5 ms·

Defer is weird in Go, being tied to function scope instead of immediate block scope.

ohCh6zos 5y ago
You're right, that is a better way to put it than what I said.
assbuttbuttass 5y ago
Function scope allows constructs like for _, lock := range locks { lock.Lock() defer lock.Unlock() } To acquire locks in a loop and release them at the end of a function. Otherwise the previous lock would be dropped each time the loop executes. I've never seen the advantage to having a block structured defer: it's easy to add a new function, it's not always possible to remove a block.
- jcelerier 5y ago
  > To acquire locks in a loop and release them at the end of a function. that looks like a super big gotcha, hasn't there been enough bugs with alloca() being called in a loop yet to show how risky and unintuitive this is ? block-scoped is explicit and explicit is good
  - kaba0 5y ago
    Good implementations of defer are scope-based, so those would be clean up at the end of the loop pretty explicitly.
- duped 5y ago
  Named scopes would alleviate this problems, and I wish more languages would adopt them.
- dzaima 5y ago
  ..Except that in C that'll turn into stack allocations and will easily segfault due to a stack overflow. So you can't actually use it like that in practice (unless you can guarantee your loop will be called a small number of times). You probably even won't notice while writing the code, and will just get random segfaults wherever you have a defer in a loop when you hit a large enough iteration count. Never mind it being very inefficient while it doesn't.
  - assbuttbuttass 5y ago
    That's a good point. I was wondering about Zig's defer, which has a block scope, and I suspect it's exactly because of this issue. Go can allocate defers on the heap, but that's a different story.
  - gmfawcett 5y ago
    Well, we are earnestly analyzing a silly little example program, but okay :). The equivalent C code wouldn't produce stack-allocated mutexes unless that's what the programmer wanted. E.g. the POSIX pthread functions don't care where your mutexes are allocated, since they are always passed by reference.
    - dzaima 5y ago
      It's not the mutexes that'd be stack-allocated, but the list of things to call back to at the end of the function. The locks list could be modified or freed by the end of the function, but something still must hold the list of things to deferred-unlock. The pthread_cleanup_push/pthread_cleanup_pop thing presumably keeps its own heap-allocated vector, backed by malloc or something. C itself can't willy nilly heap-allocate, so that list will be on the stack. But the stack is tiny compared to how long loops can be. Hence stack overflow.
      - gmfawcett 5y ago
        C libraries -- including the runtime implementations of features like this -- can heap allocate just fine. They just need to return pointers on the stack to the heap allocated values, either directly or indirectly (e.g. buried in a struct return value). As is always the case with C, the burden to free the memory is on the caller: no problem. Given a possible implementation, this loopy mutex example could have a tiny stack footprint: a single pointer to the shared cleanup() function; and a single pointer to the head of a (heap allocated) linked list of pointers to mutex (i.e., the function arguments). And the function pointer would not necessarily require allocation at all, as we can statically point at the function definition here. So we are down to a single word of stack allocation.
- gaganyaan 5y ago
  That code is odd (though I assume idiomatic Go). I'd much rather have block scoping with something like this: for _, lock := range locks { lock.Lock() // Do something with lock } And I don't really do Go, but in Rust, if I wanted to lock some arbitrary list of locks for a whole function, it would just be something like this at the top: let _ = locks.iter().map(|l| l.lock().unwrap()).collect::<Vec<_>>();
- boardwaalk 5y ago
  How often do you do something like that? Seems pretty rare. If you're managing locks maybe having an actual collection for them makes sense. If I had to choose between allowing that pattern and allowing reasonable usage in all control constructs, I'd choose the latter.
  - Someone 5y ago
    I can see one use that when using lock ordering to prevent deadlocks (http://tutorials.jenkov.com/java-concurrency/deadlock-prevention.html#ordering http://tutorials.jenkov.com/java-concurrency/deadlock-preven...), but then, the locks to be taken are a fixed set, and one probably would have function take_locks(lock *) and release_locks(lock *), and one could do defer release_locks. Also, Wikipedia (https://en.wikipedia.org/wiki/Deadlock https://en.wikipedia.org/wiki/Deadlock, https://en.wikipedia.org/wiki/Deadlock_prevention_algorithms https://en.wikipedia.org/wiki/Deadlock_prevention_algorithms) doesn’t seem to know about it. Even though I expect/guess it to be popular in embedded work, that makes me wonder whether lock ordering is used much. Could also be an omission in Wikipedia. It has https://en.wikipedia.org/wiki/Banker%27s_algorithm https://en.wikipedia.org/wiki/Banker%27s_algorithm, which I hadn’t heard of, and that Wikipedia says of “In most systems, this information is unavailable, making it impossible to implement the Banker's algorithm. Also, it is unrealistic to assume that the number of processes is static since in most systems the number of processes varies dynamically”
- KerrAvon 5y ago
  I see this code and all I can think is that you’re going to be spending the rest of your life debugging threadsafety issues. I can contrive a case where this is useful, but where in the real world?
- pcwalton 5y ago
  The semantics of block-scope defer are much simpler, easier to understand, and faster. The compiler always statically knows which defers execute at any given point, which aids optimization. With function-scope defer, the semantics are extremely dynamic, requiring bookkeeping of a runtime stack of defer thunks, and compilers have a hard time optimizing it. Function-scope defer is something that surprises everyone I explain it to. Programmers naturally expect defer to be block-scoped.
  - kgeist 5y ago
    >With function-scope defer, the semantics are extremely dynamic, requiring bookkeeping of a runtime stack of defer thunks, and compilers have a hard time optimizing it. As far as I know, most functions have only one defer, and Go optimizes such cases quite trivially, by inlining calls to the deferred function at every function exit at compile time, without managing an additional stack. If there are several defers, then yes, the slow path is used.
    - leni536 5y ago
      How does it handle a single defer in an if?
    - jibalt 5y ago
      It's very common for a function to have multiple defers ... but defers in a loop are a misuse of the feature.
  - Thaxll 5y ago
    You mixed up the two I guess?

You're right, that is a better way to put it than what I said.

Function scope allows constructs like for _, lock := range locks { lock.Lock() defer lock.Unlock() } To acquire locks in a loop and release them at the end of a function. Otherwise the previous lock would be dropped each time the loop executes. I've never seen the advantage to having a block structured defer: it's easy to add a new function, it's not always possible to remove a block.

jcelerier 5y ago
> To acquire locks in a loop and release them at the end of a function. that looks like a super big gotcha, hasn't there been enough bugs with alloca() being called in a loop yet to show how risky and unintuitive this is ? block-scoped is explicit and explicit is good
- kaba0 5y ago
  Good implementations of defer are scope-based, so those would be clean up at the end of the loop pretty explicitly.
duped 5y ago
Named scopes would alleviate this problems, and I wish more languages would adopt them.
dzaima 5y ago
..Except that in C that'll turn into stack allocations and will easily segfault due to a stack overflow. So you can't actually use it like that in practice (unless you can guarantee your loop will be called a small number of times). You probably even won't notice while writing the code, and will just get random segfaults wherever you have a defer in a loop when you hit a large enough iteration count. Never mind it being very inefficient while it doesn't.
- assbuttbuttass 5y ago
  That's a good point. I was wondering about Zig's defer, which has a block scope, and I suspect it's exactly because of this issue. Go can allocate defers on the heap, but that's a different story.
- gmfawcett 5y ago
  Well, we are earnestly analyzing a silly little example program, but okay :). The equivalent C code wouldn't produce stack-allocated mutexes unless that's what the programmer wanted. E.g. the POSIX pthread functions don't care where your mutexes are allocated, since they are always passed by reference.
  - dzaima 5y ago
    It's not the mutexes that'd be stack-allocated, but the list of things to call back to at the end of the function. The locks list could be modified or freed by the end of the function, but something still must hold the list of things to deferred-unlock. The pthread_cleanup_push/pthread_cleanup_pop thing presumably keeps its own heap-allocated vector, backed by malloc or something. C itself can't willy nilly heap-allocate, so that list will be on the stack. But the stack is tiny compared to how long loops can be. Hence stack overflow.
    - gmfawcett 5y ago
      C libraries -- including the runtime implementations of features like this -- can heap allocate just fine. They just need to return pointers on the stack to the heap allocated values, either directly or indirectly (e.g. buried in a struct return value). As is always the case with C, the burden to free the memory is on the caller: no problem. Given a possible implementation, this loopy mutex example could have a tiny stack footprint: a single pointer to the shared cleanup() function; and a single pointer to the head of a (heap allocated) linked list of pointers to mutex (i.e., the function arguments). And the function pointer would not necessarily require allocation at all, as we can statically point at the function definition here. So we are down to a single word of stack allocation.
gaganyaan 5y ago
That code is odd (though I assume idiomatic Go). I'd much rather have block scoping with something like this: for _, lock := range locks { lock.Lock() // Do something with lock } And I don't really do Go, but in Rust, if I wanted to lock some arbitrary list of locks for a whole function, it would just be something like this at the top: let _ = locks.iter().map(|l| l.lock().unwrap()).collect::<Vec<_>>();
boardwaalk 5y ago
How often do you do something like that? Seems pretty rare. If you're managing locks maybe having an actual collection for them makes sense. If I had to choose between allowing that pattern and allowing reasonable usage in all control constructs, I'd choose the latter.
- Someone 5y ago
  I can see one use that when using lock ordering to prevent deadlocks (http://tutorials.jenkov.com/java-concurrency/deadlock-prevention.html#ordering http://tutorials.jenkov.com/java-concurrency/deadlock-preven...), but then, the locks to be taken are a fixed set, and one probably would have function take_locks(lock *) and release_locks(lock *), and one could do defer release_locks. Also, Wikipedia (https://en.wikipedia.org/wiki/Deadlock https://en.wikipedia.org/wiki/Deadlock, https://en.wikipedia.org/wiki/Deadlock_prevention_algorithms https://en.wikipedia.org/wiki/Deadlock_prevention_algorithms) doesn’t seem to know about it. Even though I expect/guess it to be popular in embedded work, that makes me wonder whether lock ordering is used much. Could also be an omission in Wikipedia. It has https://en.wikipedia.org/wiki/Banker%27s_algorithm https://en.wikipedia.org/wiki/Banker%27s_algorithm, which I hadn’t heard of, and that Wikipedia says of “In most systems, this information is unavailable, making it impossible to implement the Banker's algorithm. Also, it is unrealistic to assume that the number of processes is static since in most systems the number of processes varies dynamically”
KerrAvon 5y ago
I see this code and all I can think is that you’re going to be spending the rest of your life debugging threadsafety issues. I can contrive a case where this is useful, but where in the real world?
pcwalton 5y ago
The semantics of block-scope defer are much simpler, easier to understand, and faster. The compiler always statically knows which defers execute at any given point, which aids optimization. With function-scope defer, the semantics are extremely dynamic, requiring bookkeeping of a runtime stack of defer thunks, and compilers have a hard time optimizing it. Function-scope defer is something that surprises everyone I explain it to. Programmers naturally expect defer to be block-scoped.
- kgeist 5y ago
  >With function-scope defer, the semantics are extremely dynamic, requiring bookkeeping of a runtime stack of defer thunks, and compilers have a hard time optimizing it. As far as I know, most functions have only one defer, and Go optimizes such cases quite trivially, by inlining calls to the deferred function at every function exit at compile time, without managing an additional stack. If there are several defers, then yes, the slow path is used.
  - leni536 5y ago
    How does it handle a single defer in an if?
  - jibalt 5y ago
    It's very common for a function to have multiple defers ... but defers in a loop are a misuse of the feature.
- Thaxll 5y ago
  You mixed up the two I guess?