5 ms·

I don't understand the intuition of closures and they turn me off to languages immediately. They feel like a hack from someone who didn't want to store a copy o

I don't understand the intuition of closures and they turn me off to languages immediately. They feel like a hack from someone who didn't want to store a copy of a parent-scope variable within a function. The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd) makes me feel that it is impossible to reason about variable lifetimes when dealing with closures. Is there some perspective I'm missing here? Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function?

merlincorey 7y ago
There are indeed some nuances you are missing as well as at least one misconception you are holding: > The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd) You can't touch variables that have been garbage collected, indeed. A variable captured in a closure will have a reference to it and therefore not be garbage collected until the closure itself goes out of scope and is garbage collected. Closures allow encoding some state into a function, which is especially useful for higher order functions that return specialized functions, for example. This is, for example, how decorators in Python work - a function is captured in a closure, which is then referenced and called by a new function wrapping the original function.
chubot 7y ago
I don't have a problem with the intuition but I'm with you on programming style. It's sort of a contrary opinion but I prefer to be explicit about state, and closures are sort of silently bundling up state for you behind the scenes. It's not very apparent from the syntax. I think languages should have a lighter-weight syntax for classes and that would subsume many use cases for closures. ----- While I've never programmed in PHP, it appears PHP actually has a pretty nice solution for this problem! You declare the variables captured using 'use': https://stackoverflow.com/questions/1065188/in-php-what-is-a-closure-and-why-does-it-use-the-use-identifier https://stackoverflow.com/questions/1065188/in-php-what-is-a... $callback = function ($quantity, $product) use ($tax, &$total) In other languages the capture of 'tax' and 'total' are implicit.
- smilekzs 7y ago
  C++ lambda has explicit capture list as well.
- kragen 7y ago
  Yeah, I came to the same conclusion about closures when I wrote Ur-Scheme. In mainstream modern languages, you have the remarkable situation that, when you are writing the code, you know which variables you intend your closure to capture, but you don't note that explicitly in the code. Then the compiler needs some extra complexity to calculate the set of free variables in the body of the lambda — and which binding contours they are being captured from, although that's usually pretty trivial. And then the guy who is changing the code next year also needs to reverse-engineer that same set of variables and binding contours in order to modify it successfully. So it seems sort of perverse to leave that information out of the code! Since then, though, I've come to doubt my conclusion, for two reasons: 1. The same thing is true of, for example, static types; but using type inference or dynamic typing often makes your code easier rather than harder to read. (Some wag said something to the effect that dynamic typing is what you do when it's simultaneously so trivial to see that your program has no type errors that you can do it in your head and so difficult that you don't want to spend the time to do the proof.) 2. There isn't a really compelling difference to me between the closure in ^(x < 2 ifTrue: [ 1 ] ifFalse: [ (x - 1) factorial ]) capturing x and the closure in fetch(json).then(r => widget.displayJson(r.json())) capturing widget, or the block in (let ((overlay (make-overlay beg end))) (overlay-put overlay 'face (or face 'highlight))) capturing face. That is, inner blocks of control structures implicitly capture variables from their outer scope all the time, and this is mostly not a problem; and closures are a useful technique to make it possible to extend the set of control structures. (As it happens, let in Emacs Lisp is implemented as a built-in special form, but in Scheme it's normally implemented as a macro that puts the inner block into a closure.) Maybe part of the reason is the lifetime: the example with fetch() is in fact socking away a reference to widget until after the HTTP response comes back, so aliasing bugs and space leaks are possible, while the other two examples aren't. This might be a reason implicit closures are so much more popular in functional programming languages: aliasing bugs are not a problem for immutable data, and nobody expects to be able to predict how much memory their functional program will need anyway.
wffurr 7y ago
In the article they discuss that Rust closures are just sugar for structs that work exactly as you describe. As with any other syntactic sugar, it's easy to reason about the behavior once you know what the syntax desugars into. One classic example I see with beginning programmers all the time is the order of the parts of a for loop. e.g. for(initialization statement; bounds check expression; increment statement) { ... } is just syntactic sugar for: { initialization statement; while (bounds check expression) { ... increment statement; } } But once you understand what a for-loop desugars into, it's trivial to use and reason about. Closures aren't much different. In Rust, if anything, they're less magic, because you have to be explicit about the lifetime of the closure relative to the references it's borrowing.
- cesarb 7y ago
  > is just syntactic sugar for [...] There's just a subtle detail to keep in mind here: both "continue" and "break" are syntactic sugar for "goto", and when converting a "for" loop into a "while" loop, you have to also convert "continue" to a "goto" to a label just before the increment statement.
deleted 7y ago
[deleted]
kxyvr 7y ago
In many ways, closures can be more difficult when there is not garbage collection because we precisely do not want to refer to values that have been freed or collected. For example, in C++, we can write code like: #include <functional> #include <iostream> auto foo(double a) -> std::function <double(double)> { return [&a](auto x) { return x + a; }; } int main() { auto f = foo(2.0); std::cout << f(1.0) << std::endl; std::cout << f(1.0) << std::endl; } This code is bad because the function returned by `foo` depends on a reference to `a`, which is freed at the end of `foo`. Running this gives: 3 1 which is bad. Rust has the advantage of finding such errors. For example, the following Rust code: fn foo(a: f64) -> impl Fn(f64) -> f64 { |x: f64| -> f64 { x + a } } fn main() { let f = foo(2.0); println!("{}", f(1.0)); println!("{}", f(1.0)); } fails to compile with the error: error[E0597]: `a` does not live long enough --> test01.rs:2:27 | 1 | fn foo(a: f64) -> impl Fn(f64) -> f64 { | ------------------- opaque type requires that `a` is borrowed for `'static` 2 | |x: f64| -> f64 { x + a } | --------------- ^ borrowed value does not live long enough | | | value captured here 3 | } | - `a` dropped here while still borrowed That forces us to correct the code by adding the move annotation to the the closure, which moves ownership of a to the closure itself: fn foo(a: f64) -> impl Fn(f64) -> f64 { move |x: f64| -> f64 { x + a } } fn main() { let f = foo(2.0); println!("{}", f(1.0)); println!("{}", f(1.0)); } This gives the correct result: 3 3 Of course, all of this nuance can be avoided in a garbage collected language. For example, in OCaml, we can write: let foo a = fun x -> x +. a in let f = foo 2.0 in Printf.printf "%f\n" (f 1.0); Printf.printf "%f\n" (f 1.0); which also returns the correct result: 3.000000 3.000000 More generally, closures are nice because it's a useful feature to return a function from a function. For example, we may want call a function parameterized on some parameters repeatedly and we don't want to have to pass all of the parameters each time we invoke the function. Functional programming and closures allow for a variety of other programming techniques as well. Beyond that, my point is that a garbage collector can track the use of variables and memory extremely well, which makes the process of constructing closures moderately easy. We can have closures without garbage collection, but then we must be careful about who and what owns this memory. C++ does not enforce this, but Rust does, which is one of its benefits.
sdegutis 7y ago
In JavaScript and similar languages, “yes.” It’s really a lot simpler than you’re making it. Closures are great in these languages and incredibly useful for rapid prototyping. Especially in Node.js and in browsers.
- dnautics 7y ago
  not just rapid prototyping. Often times, when i write unit tests and the like, i use closures. In the system i work in, asserting an exception requires spawning a new thread and then watching it burn, which is most easily accomplished with a closure. But also writing one-off tasks to be run in a new thread (even in prod) can take closures.
mrkeen 7y ago
> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function? I believe that would be 'lambda-lifting', i.e. taking a lambda which has free variables, adding those free variables to its input variables, moving the lambda out into the top- level, and modifying the call site to pass in the no-longer-free-variables. That should give you completely reasonable lifetimes and only require the stack. It gets trickier when you want functions to be able to return functions. For instance: add(a, b) { a + b } myFun() { var add5 = x -> add(5, x); var add6 = x -> add(6, x); add5(1) + add6(2) } In this case I've made two variants of add. I can't just run add5 on the stack, get rid of it, and then run add6 afterwards, because they both need to be alive at the same time.
ufo 7y ago
> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function? Pretty much. When implementing closures, the variables that are going to be used by the nested functions are allocated on the heap instead of on the stack. The nested function is then set up to receive a pointer to the outer variables as an additional parameter. If you want to read more about it, the word to search for would be "closure conversion" http://matt.might.net/articles/closure-conversion/ http://matt.might.net/articles/closure-conversion/ It's unfortunate the the Wikipedia article for closure conversion now redirects to the article about Lambda Lifting. They are related but aren't quite the same thing.
saghm 7y ago
> The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd) I think this might be due to your mental model of GC not quite matching the way it's usually implemented (at least, in traditional GC, as opposed to reference counting); most GC gives no guarantees that memory will immediately be cleaned up once the last reference goes out of scope, just that it will happen some time in the future (usually the next time it checks). Languages like Rust (and C++, if you use destructors) will inline the memory cleanup at the end of the scope, but this is pretty explicitly not what will happen in a garbage-collected language. Note also that this isn't really any different than the fact that a variable can go out of scope and not be cleaned up if another reference to it exists. Consider the following Java-esque code: void setNewChild(Parent parent) { var child = new Child(); parent.child = child; } The `Child` created in the method body will not be garbage collected even though the variable goes out of scope because it's still referenced somewhere else (i.e. in the Parent object). The key insight to understanding closures is that using the local variable in another function makes another reference to it, so naturally it would be kept alive until that function is gone.
- Sean1708 7y ago
  > most GC gives no guarantees that memory will immediately be cleaned up once the last reference goes out of scope But in this example the last reference hasn't gone out of scope, the last reference is in the closure. If the GC ignored closures and acted as if last reference had gone out of scope then there's a chance that that variable would be cleaned up before the closure gets called, which would make closures completely useless.
  - saghm 7y ago
    I think we're saying the same thing! The point I was trying to make with the second half of my comment was that a variable going out of scope isn't the same thing as the object going out scope, since other references can still be alive afterwards. The example I gave was intended to demonstrate this without using closures (since the person I was responding to had indicated that they found them confusing), and then went on to attempt to explain that the example with just objects was in fact doing the same thing as if the object were referenced in a closure rather than another object. The first half of my comment was just an attempt to respond to the statement about the object having "ostensibly been GC'd", which isn't necessarily the case even if there wasn't a closure referencing the object; in your phrasing, there's a chance that it would be cleaned up, and I wanted to make sure that it was clear that it was only a chance, not a guarantee.
samatman 7y ago
I've got just the paper for you! The Implementation of Closures in Lua: https://pdfs.semanticscholar.org/73a2/e3c03f799956aa5a3188e4eb35c90977a471.pdf https://pdfs.semanticscholar.org/73a2/e3c03f799956aa5a3188e4... This is just one way to do it, but it's an efficient way, and easy to understand.
wwright 7y ago
I know you've already got a storm of replies, but I'd like to illustrate part of what I love about closure (especially when combined with anonymous objects like in JavaScript): you can replace classes, private vs public, "this", and other features with a simpler, smaller number of features. const Counter = () => { let counts = new Map(); return { reset: () => { counts = new Map(); }, count: (key) => { const count = counts.get(key) + 1; counts.set(key, count); return count; } } } const items = ['apple', 'apple', 'banana', 'canteloupe']; const counter = Counter() items.forEach(item => { console.log(item + ": count is", counter.count(item)); }) counter.reset(); // do some more counting now IMO, this is much simpler [and prettier ;)] than the alternative using classes: you only need to know closures and objects, and the rules apply the same as they do in all other contexts. Classes in most languages usually come with their own twists and surprises.
zozbot234 7y ago
> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function? Yup. A "first class" closure is just reifying some arguments of a future function call and bundling them up nicely in a struct/record. This is especially clear in languages that support currying, since that means you'll literally be building that record step-by-step - only after a full set of arguments is provided does a true function call really occur.

merlincorey 7y ago

There are indeed some nuances you are missing as well as at least one misconception you are holding: > The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd) You can't touch variables that have been garbage collected, indeed. A variable captured in a closure will have a reference to it and therefore not be garbage collected until the closure itself goes out of scope and is garbage collected. Closures allow encoding some state into a function, which is especially useful for higher order functions that return specialized functions, for example. This is, for example, how decorators in Python work - a function is captured in a closure, which is then referenced and called by a new function wrapping the original function.

chubot 7y ago

I don't have a problem with the intuition but I'm with you on programming style. It's sort of a contrary opinion but I prefer to be explicit about state, and closures are sort of silently bundling up state for you behind the scenes. It's not very apparent from the syntax. I think languages should have a lighter-weight syntax for classes and that would subsume many use cases for closures. ----- While I've never programmed in PHP, it appears PHP actually has a pretty nice solution for this problem! You declare the variables captured using 'use': https://stackoverflow.com/questions/1065188/in-php-what-is-a-closure-and-why-does-it-use-the-use-identifier https://stackoverflow.com/questions/1065188/in-php-what-is-a... $callback = function ($quantity, $product) use ($tax, &$total) In other languages the capture of 'tax' and 'total' are implicit.

smilekzs 7y ago
C++ lambda has explicit capture list as well.
kragen 7y ago
Yeah, I came to the same conclusion about closures when I wrote Ur-Scheme. In mainstream modern languages, you have the remarkable situation that, when you are writing the code, you know which variables you intend your closure to capture, but you don't note that explicitly in the code. Then the compiler needs some extra complexity to calculate the set of free variables in the body of the lambda — and which binding contours they are being captured from, although that's usually pretty trivial. And then the guy who is changing the code next year also needs to reverse-engineer that same set of variables and binding contours in order to modify it successfully. So it seems sort of perverse to leave that information out of the code! Since then, though, I've come to doubt my conclusion, for two reasons: 1. The same thing is true of, for example, static types; but using type inference or dynamic typing often makes your code easier rather than harder to read. (Some wag said something to the effect that dynamic typing is what you do when it's simultaneously so trivial to see that your program has no type errors that you can do it in your head and so difficult that you don't want to spend the time to do the proof.) 2. There isn't a really compelling difference to me between the closure in ^(x < 2 ifTrue: [ 1 ] ifFalse: [ (x - 1) factorial ]) capturing x and the closure in fetch(json).then(r => widget.displayJson(r.json())) capturing widget, or the block in (let ((overlay (make-overlay beg end))) (overlay-put overlay 'face (or face 'highlight))) capturing face. That is, inner blocks of control structures implicitly capture variables from their outer scope all the time, and this is mostly not a problem; and closures are a useful technique to make it possible to extend the set of control structures. (As it happens, let in Emacs Lisp is implemented as a built-in special form, but in Scheme it's normally implemented as a macro that puts the inner block into a closure.) Maybe part of the reason is the lifetime: the example with fetch() is in fact socking away a reference to widget until after the HTTP response comes back, so aliasing bugs and space leaks are possible, while the other two examples aren't. This might be a reason implicit closures are so much more popular in functional programming languages: aliasing bugs are not a problem for immutable data, and nobody expects to be able to predict how much memory their functional program will need anyway.

wffurr 7y ago

In the article they discuss that Rust closures are just sugar for structs that work exactly as you describe. As with any other syntactic sugar, it's easy to reason about the behavior once you know what the syntax desugars into. One classic example I see with beginning programmers all the time is the order of the parts of a for loop. e.g. for(initialization statement; bounds check expression; increment statement) { ... } is just syntactic sugar for: { initialization statement; while (bounds check expression) { ... increment statement; } } But once you understand what a for-loop desugars into, it's trivial to use and reason about. Closures aren't much different. In Rust, if anything, they're less magic, because you have to be explicit about the lifetime of the closure relative to the references it's borrowing.

cesarb 7y ago
> is just syntactic sugar for [...] There's just a subtle detail to keep in mind here: both "continue" and "break" are syntactic sugar for "goto", and when converting a "for" loop into a "while" loop, you have to also convert "continue" to a "goto" to a label just before the increment statement.

deleted 7y ago

[deleted]

kxyvr 7y ago

In many ways, closures can be more difficult when there is not garbage collection because we precisely do not want to refer to values that have been freed or collected. For example, in C++, we can write code like: #include <functional> #include <iostream> auto foo(double a) -> std::function <double(double)> { return [&a](auto x) { return x + a; }; } int main() { auto f = foo(2.0); std::cout << f(1.0) << std::endl; std::cout << f(1.0) << std::endl; } This code is bad because the function returned by `foo` depends on a reference to `a`, which is freed at the end of `foo`. Running this gives: 3 1 which is bad. Rust has the advantage of finding such errors. For example, the following Rust code: fn foo(a: f64) -> impl Fn(f64) -> f64 { |x: f64| -> f64 { x + a } } fn main() { let f = foo(2.0); println!("{}", f(1.0)); println!("{}", f(1.0)); } fails to compile with the error: error[E0597]: `a` does not live long enough --> test01.rs:2:27 | 1 | fn foo(a: f64) -> impl Fn(f64) -> f64 { | ------------------- opaque type requires that `a` is borrowed for `'static` 2 | |x: f64| -> f64 { x + a } | --------------- ^ borrowed value does not live long enough | | | value captured here 3 | } | - `a` dropped here while still borrowed That forces us to correct the code by adding the move annotation to the the closure, which moves ownership of a to the closure itself: fn foo(a: f64) -> impl Fn(f64) -> f64 { move |x: f64| -> f64 { x + a } } fn main() { let f = foo(2.0); println!("{}", f(1.0)); println!("{}", f(1.0)); } This gives the correct result: 3 3 Of course, all of this nuance can be avoided in a garbage collected language. For example, in OCaml, we can write: let foo a = fun x -> x +. a in let f = foo 2.0 in Printf.printf "%f\n" (f 1.0); Printf.printf "%f\n" (f 1.0); which also returns the correct result: 3.000000 3.000000 More generally, closures are nice because it's a useful feature to return a function from a function. For example, we may want call a function parameterized on some parameters repeatedly and we don't want to have to pass all of the parameters each time we invoke the function. Functional programming and closures allow for a variety of other programming techniques as well. Beyond that, my point is that a garbage collector can track the use of variables and memory extremely well, which makes the process of constructing closures moderately easy. We can have closures without garbage collection, but then we must be careful about who and what owns this memory. C++ does not enforce this, but Rust does, which is one of its benefits.

sdegutis 7y ago

In JavaScript and similar languages, “yes.” It’s really a lot simpler than you’re making it. Closures are great in these languages and incredibly useful for rapid prototyping. Especially in Node.js and in browsers.

dnautics 7y ago
not just rapid prototyping. Often times, when i write unit tests and the like, i use closures. In the system i work in, asserting an exception requires spawning a new thread and then watching it burn, which is most easily accomplished with a closure. But also writing one-off tasks to be run in a new thread (even in prod) can take closures.

mrkeen 7y ago

> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function? I believe that would be 'lambda-lifting', i.e. taking a lambda which has free variables, adding those free variables to its input variables, moving the lambda out into the top- level, and modifying the call site to pass in the no-longer-free-variables. That should give you completely reasonable lifetimes and only require the stack. It gets trickier when you want functions to be able to return functions. For instance: add(a, b) { a + b } myFun() { var add5 = x -> add(5, x); var add6 = x -> add(6, x); add5(1) + add6(2) } In this case I've made two variants of add. I can't just run add5 on the stack, get rid of it, and then run add6 afterwards, because they both need to be alive at the same time.

ufo 7y ago

> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function? Pretty much. When implementing closures, the variables that are going to be used by the nested functions are allocated on the heap instead of on the stack. The nested function is then set up to receive a pointer to the outer variables as an additional parameter. If you want to read more about it, the word to search for would be "closure conversion" http://matt.might.net/articles/closure-conversion/ http://matt.might.net/articles/closure-conversion/ It's unfortunate the the Wikipedia article for closure conversion now redirects to the article about Lambda Lifting. They are related but aren't quite the same thing.

saghm 7y ago

> The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd) I think this might be due to your mental model of GC not quite matching the way it's usually implemented (at least, in traditional GC, as opposed to reference counting); most GC gives no guarantees that memory will immediately be cleaned up once the last reference goes out of scope, just that it will happen some time in the future (usually the next time it checks). Languages like Rust (and C++, if you use destructors) will inline the memory cleanup at the end of the scope, but this is pretty explicitly not what will happen in a garbage-collected language. Note also that this isn't really any different than the fact that a variable can go out of scope and not be cleaned up if another reference to it exists. Consider the following Java-esque code: void setNewChild(Parent parent) { var child = new Child(); parent.child = child; } The `Child` created in the method body will not be garbage collected even though the variable goes out of scope because it's still referenced somewhere else (i.e. in the Parent object). The key insight to understanding closures is that using the local variable in another function makes another reference to it, so naturally it would be kept alive until that function is gone.

Sean1708 7y ago
> most GC gives no guarantees that memory will immediately be cleaned up once the last reference goes out of scope But in this example the last reference hasn't gone out of scope, the last reference is in the closure. If the GC ignored closures and acted as if last reference had gone out of scope then there's a chance that that variable would be cleaned up before the closure gets called, which would make closures completely useless.
- saghm 7y ago
  I think we're saying the same thing! The point I was trying to make with the second half of my comment was that a variable going out of scope isn't the same thing as the object going out scope, since other references can still be alive afterwards. The example I gave was intended to demonstrate this without using closures (since the person I was responding to had indicated that they found them confusing), and then went on to attempt to explain that the example with just objects was in fact doing the same thing as if the object were referenced in a closure rather than another object. The first half of my comment was just an attempt to respond to the statement about the object having "ostensibly been GC'd", which isn't necessarily the case even if there wasn't a closure referencing the object; in your phrasing, there's a chance that it would be cleaned up, and I wanted to make sure that it was clear that it was only a chance, not a guarantee.

samatman 7y ago

I've got just the paper for you! The Implementation of Closures in Lua: https://pdfs.semanticscholar.org/73a2/e3c03f799956aa5a3188e4eb35c90977a471.pdf https://pdfs.semanticscholar.org/73a2/e3c03f799956aa5a3188e4... This is just one way to do it, but it's an efficient way, and easy to understand.

wwright 7y ago

I know you've already got a storm of replies, but I'd like to illustrate part of what I love about closure (especially when combined with anonymous objects like in JavaScript): you can replace classes, private vs public, "this", and other features with a simpler, smaller number of features. const Counter = () => { let counts = new Map(); return { reset: () => { counts = new Map(); }, count: (key) => { const count = counts.get(key) + 1; counts.set(key, count); return count; } } } const items = ['apple', 'apple', 'banana', 'canteloupe']; const counter = Counter() items.forEach(item => { console.log(item + ": count is", counter.count(item)); }) counter.reset(); // do some more counting now IMO, this is much simpler [and prettier ;)] than the alternative using classes: you only need to know closures and objects, and the rules apply the same as they do in all other contexts. Classes in most languages usually come with their own twists and surprises.

zozbot234 7y ago

> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function? Yup. A "first class" closure is just reifying some arguments of a future function call and bundling them up nicely in a struct/record. This is especially clear in languages that support currying, since that means you'll literally be building that record step-by-step - only after a full set of arguments is provided does a true function call really occur.