6 ms·

This is very very long to explain a very simple pair of ideas: 1) You should be able to declare scoped blocks that mandate execution of all tasks started in th

This is very very long to explain a very simple pair of ideas: 1) You should be able to declare scoped blocks that mandate execution of all tasks started in that block ends when the scope ends. 2) This is fundamentally superior to all other forms of concurrency. I get it; this is basically what async/await gets you, but conceptually you can spawn parallel tasks inside an awaited block, and know, absolutely that all of those tasks are resolved when the await resolves. (this is distinct from a normal awaited block which will execute tasks inside it sequentially, awaiting each one in turn). ...seems like an interesting (and novel) idea to me, but I flat our reject (2) as ridiculous. Parallel programming is hard, but the approach from rust, to give you formal verification, instead of arbitrarily throwing away useful tools seems much more realistic to me.

munjal116 8y ago
Or Promise.all/Promise.reject for that matter.
pcwalton 8y ago
Pure fork/join concurrency isn't a novel idea. I was doing some work on it in grad school. Without blocking channels, fork/join has some really nice properties, such as guaranteed deadlock freedom. However, I agree with you that there's no one-size-fits-all approach to concurrency. Sometimes you want long-lived tasks that communicate—the actor model, in other words—in which case fork/join doesn't buy you much. I do think that fork/join is frequently what you want, though.
joshuamorton 8y ago
>this is distinct from a normal awaited block which will execute tasks inside it sequentially, awaiting each one in turn I believe that python's `async for` construct (which may be stolen from c#, but I'm not sure) does this, and another user mentioned Promise.all which in javascript I believe allows the child promises to resolve concurrently. In both cases, these are for concurrency, not parallelism.
- wokwokwok 8y ago
  The point of the construct is not explicitly waiting for child tasks/threads/whatever, but that as a formal restriction, you cannot leave the scoped block with unresolved tasks. I don't believe either of the constructs you've mentioned do this, and I'm not aware of any that do. The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code. It doesn't matter if you've called Promise.all or not. Also, regarding parallelism: Obviously async/await are for that; this is basically pitching an equivalent construct for parallel processing. I think this is novel, frankly, and the alternatives being thrown around by people are by people who didn't read the article.
  - joshuamorton 8y ago
    >The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code. Well of course, setTimeout isn't async/await based, its callback based. If instead you only had this api, you could make that formal assertion: await promiseSetTimeout(time).then(...); And in fact you can write promiseSetTimeout today! So basically, async/await provides this as long as you only use async/await. In JS you can't formally assert this because there are functions that subvert the normal control flow, but if you outlaw those, you absolutely can.
    - wokwokwok 8y ago
      > as long as you only use async/await... Well, what if you don't? That's the point. The proposed construct does not require that: I'm not saying its better or worse; I'm just pointing out that it's different. Obviously if you choose to avoid branching code in your functions, they won't branch; what's novel here is that the closing scope (ie. in this case, the collapse of the python with block) triggers a collection of all the ambient tasks started in that context. I'm sure you could implement something similar in javascript, but it is not the same as Promise.all.
      - joshuamorton 8y ago
        >Well, what if you don't? This is kind of a silly question. Its always possible to subvert a safe construct system if you try hard enough. You can write unsafe blocks in rust. You can pass in a callback to a nursery, as is described in the article. >The proposed construct does not require that Well, kind of. The article actually explicitly states that >Here's a simpler primitive that would also satisfy our flow control diagram above. It takes a list of thunks, and runs them all concurrently: which is equivalent to promise.all, satisfies the invariant nurseries do, except in very specific circumstances (unbounded while loop-y constructs). And you just simply can't use promise.all in that situation.
pacala 8y ago
+1. This is structured programming, applied to concurrent setting. The argument for structured programming was made and won in the '60s [0]. It is amazing that we keep making the same mistakes over and over again. > The unbridled use of the go to statement has an immediate consequence that it becomes terribly hard to find a meaningful set of coordinates in which to describe the process progress. Usually, people take into account as well the values of some well chosen variables, but this is out of the question because it is relative to the progress that the meaning of these values is to be understood! With the go to statement one can, of course, still describe the progress uniquely by a counter counting the number of actions performed since program start (viz. a kind of normalized clock). The difficulty is that such a coordinate, although unique, is utterly unhelpful. In such a coordinate system it becomes an extremely complicated affair to define all those points of progress where, say, n equals the number of persons in the room minus one! [0] http://www.u.arizona.edu/~rubinson/copyright_violations/Go_To_Considered_Harmful.html http://www.u.arizona.edu/~rubinson/copyright_violations/Go_T...
- cwzwarich 8y ago
  > The argument for structured programming was made and won in the '60s Most code out there is fine using break and continue in loops, and early returns are pretty popular. All of these are unstructured programming (by the 60s definition). I don't think it's all that clear that it has won.
  - scott_s 8y ago
    The author anticipates this point, and rebuts it: "In the end, modern languages are a bit less strict about this than Dijkstra's original formulation. They'll let you break out of multiple nested structures at once using constructs like break, continue, or return. But fundamentally, they're all designed around Dijkstra's idea; even these constructs that push the boundaries do so only in strictly limited ways. In particular, functions – which are the fundamental tool for wrapping up control flow inside a black box – are considered inviolate. You can't break out of one function and into another, and a return can take you out of the current function, but no further. Whatever control flow shenanigans a function gets up to internally, other functions don't have to care. This even extends to goto itself. You'll find a few languages that still have something they call goto, like C, C#, Golang, ... but they've added heavy restrictions. At the very least, they won't let you jump out of one function body and into another. Unless you're working in assembly, the classic, unrestricted goto is gone. Dijkstra won." I tend to agree with the author. Linux kernel code, for example, uses goto for error handling, particularly when implementing system calls. But the author is correct: the gotos can't jump out of the function. Readers can still infer control flow from function call sequences. I quite enjoyed the post, I think it's worth giving the author's claims serious consideration.
appleflaxen 8y ago
The article makes a strong case. Even if rust allows formal verification, not all programmers use it, and if you use their library, you don't know if it's been verified. Take it back to the goto analogy: can you formally verify goto? Yes. Does that mean it's good to include as a language primitive? No. You are basically taking an entire argument, ignoring its merits, and saying "but you can do it another way". You are exactly right, but you haven't rebutted the fact that structured concurrency is philosophically superior. I love rust and the community; they will get to the truth of this argument eventually. But I suspect the truth is that "njs was right", and I hope the it's sooner rather than later.
- wilun 8y ago
  It's not only that Rust allows formal verification, it's that it does it by default, writing unsafe Rust code for no serious reason is fundamentally frowned upon, and even then there is a serious effort to bring some tooling to bring more confidence even on "unsafe" Rust code. > But I suspect the truth is that "njs was right", and I hope the it's sooner rather than later. I'm not sure. The problem can and has been solved without the manually "pass the nursery object around" ""escape hatch"" for the general case of threads (because no: having the execution of spawning functions delayed by the lifetime of thread is arguably reasonable is some cases, but certainly not the general case of what threads are useful and used for) It is still useful for tons of existing languages as a pattern anyway, but only if the use cases are suitable. But the author is focused on a narrow use case of threads (and a narrow subset of the problems they introduce), and present their solution as a general truth and new fundamental control structure of computing, independent of already existing, in production, and arguably better solutions; and independent of analyzing the new problems their silver bullet introduces.
  - iainmerrick 8y ago
    “Formal verification” for Rust code -- does that actually exist yet, or do people just hope/assume that the language will be proven consistent and that awesome theorem-checking tools will emerge eventually? I agree that Rust has a great model that seems to lead to very solid code, but “formal verification” is a high bar to clear.
wilun 8y ago
I found it still an interesting read, but I agree with you. The most important problem is: > Then our guarantee is lost: the operations that look like they're inside the with block might actually keep running after the with block ends, and then crash because the file gets closed while they're still using it. And again, you can't tell from local inspection; to know if this is happening you have to go read the source code to all the functions called inside the ... code. But actually you can "tell", or even better have a type system good enough to prevent such mistakes entirely, and the author even knows a bit about Rust, yet fails to state that at least that item is a solved problem there (by using a different and arguably more general approach). Now I don't know enough to decide whether something is missing on the panicking background thread front, but if it does that seems very solvable. I don't buy that spawning threads (or even moral equivalents) and multiprogrammation is in a situation similar to unstructured use of goto anyway. You have tons of problems applicable to one and not the other. And of course resource management is hard to get right with threads. But we have at least a production example of a language that get it right on some points by leveraging more general ideas (and the other difficult points are mostly not addressed by the nursery idea, anyway).

munjal116 8y ago

Or Promise.all/Promise.reject for that matter.

pcwalton 8y ago

Pure fork/join concurrency isn't a novel idea. I was doing some work on it in grad school. Without blocking channels, fork/join has some really nice properties, such as guaranteed deadlock freedom. However, I agree with you that there's no one-size-fits-all approach to concurrency. Sometimes you want long-lived tasks that communicate—the actor model, in other words—in which case fork/join doesn't buy you much. I do think that fork/join is frequently what you want, though.

joshuamorton 8y ago

>this is distinct from a normal awaited block which will execute tasks inside it sequentially, awaiting each one in turn I believe that python's `async for` construct (which may be stolen from c#, but I'm not sure) does this, and another user mentioned Promise.all which in javascript I believe allows the child promises to resolve concurrently. In both cases, these are for concurrency, not parallelism.

wokwokwok 8y ago
The point of the construct is not explicitly waiting for child tasks/threads/whatever, but that as a formal restriction, you cannot leave the scoped block with unresolved tasks. I don't believe either of the constructs you've mentioned do this, and I'm not aware of any that do. The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code. It doesn't matter if you've called Promise.all or not. Also, regarding parallelism: Obviously async/await are for that; this is basically pitching an equivalent construct for parallel processing. I think this is novel, frankly, and the alternatives being thrown around by people are by people who didn't read the article.
- joshuamorton 8y ago
  >The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code. Well of course, setTimeout isn't async/await based, its callback based. If instead you only had this api, you could make that formal assertion: await promiseSetTimeout(time).then(...); And in fact you can write promiseSetTimeout today! So basically, async/await provides this as long as you only use async/await. In JS you can't formally assert this because there are functions that subvert the normal control flow, but if you outlaw those, you absolutely can.
  - wokwokwok 8y ago
    > as long as you only use async/await... Well, what if you don't? That's the point. The proposed construct does not require that: I'm not saying its better or worse; I'm just pointing out that it's different. Obviously if you choose to avoid branching code in your functions, they won't branch; what's novel here is that the closing scope (ie. in this case, the collapse of the python with block) triggers a collection of all the ambient tasks started in that context. I'm sure you could implement something similar in javascript, but it is not the same as Promise.all.
    - joshuamorton 8y ago
      >Well, what if you don't? This is kind of a silly question. Its always possible to subvert a safe construct system if you try hard enough. You can write unsafe blocks in rust. You can pass in a callback to a nursery, as is described in the article. >The proposed construct does not require that Well, kind of. The article actually explicitly states that >Here's a simpler primitive that would also satisfy our flow control diagram above. It takes a list of thunks, and runs them all concurrently: which is equivalent to promise.all, satisfies the invariant nurseries do, except in very specific circumstances (unbounded while loop-y constructs). And you just simply can't use promise.all in that situation.

pacala 8y ago

+1. This is structured programming, applied to concurrent setting. The argument for structured programming was made and won in the '60s [0]. It is amazing that we keep making the same mistakes over and over again. > The unbridled use of the go to statement has an immediate consequence that it becomes terribly hard to find a meaningful set of coordinates in which to describe the process progress. Usually, people take into account as well the values of some well chosen variables, but this is out of the question because it is relative to the progress that the meaning of these values is to be understood! With the go to statement one can, of course, still describe the progress uniquely by a counter counting the number of actions performed since program start (viz. a kind of normalized clock). The difficulty is that such a coordinate, although unique, is utterly unhelpful. In such a coordinate system it becomes an extremely complicated affair to define all those points of progress where, say, n equals the number of persons in the room minus one! [0] http://www.u.arizona.edu/~rubinson/copyright_violations/Go_To_Considered_Harmful.html http://www.u.arizona.edu/~rubinson/copyright_violations/Go_T...

cwzwarich 8y ago
> The argument for structured programming was made and won in the '60s Most code out there is fine using break and continue in loops, and early returns are pretty popular. All of these are unstructured programming (by the 60s definition). I don't think it's all that clear that it has won.
- scott_s 8y ago
  The author anticipates this point, and rebuts it: "In the end, modern languages are a bit less strict about this than Dijkstra's original formulation. They'll let you break out of multiple nested structures at once using constructs like break, continue, or return. But fundamentally, they're all designed around Dijkstra's idea; even these constructs that push the boundaries do so only in strictly limited ways. In particular, functions – which are the fundamental tool for wrapping up control flow inside a black box – are considered inviolate. You can't break out of one function and into another, and a return can take you out of the current function, but no further. Whatever control flow shenanigans a function gets up to internally, other functions don't have to care. This even extends to goto itself. You'll find a few languages that still have something they call goto, like C, C#, Golang, ... but they've added heavy restrictions. At the very least, they won't let you jump out of one function body and into another. Unless you're working in assembly, the classic, unrestricted goto is gone. Dijkstra won." I tend to agree with the author. Linux kernel code, for example, uses goto for error handling, particularly when implementing system calls. But the author is correct: the gotos can't jump out of the function. Readers can still infer control flow from function call sequences. I quite enjoyed the post, I think it's worth giving the author's claims serious consideration.

appleflaxen 8y ago

The article makes a strong case. Even if rust allows formal verification, not all programmers use it, and if you use their library, you don't know if it's been verified. Take it back to the goto analogy: can you formally verify goto? Yes. Does that mean it's good to include as a language primitive? No. You are basically taking an entire argument, ignoring its merits, and saying "but you can do it another way". You are exactly right, but you haven't rebutted the fact that structured concurrency is philosophically superior. I love rust and the community; they will get to the truth of this argument eventually. But I suspect the truth is that "njs was right", and I hope the it's sooner rather than later.

wilun 8y ago
It's not only that Rust allows formal verification, it's that it does it by default, writing unsafe Rust code for no serious reason is fundamentally frowned upon, and even then there is a serious effort to bring some tooling to bring more confidence even on "unsafe" Rust code. > But I suspect the truth is that "njs was right", and I hope the it's sooner rather than later. I'm not sure. The problem can and has been solved without the manually "pass the nursery object around" ""escape hatch"" for the general case of threads (because no: having the execution of spawning functions delayed by the lifetime of thread is arguably reasonable is some cases, but certainly not the general case of what threads are useful and used for) It is still useful for tons of existing languages as a pattern anyway, but only if the use cases are suitable. But the author is focused on a narrow use case of threads (and a narrow subset of the problems they introduce), and present their solution as a general truth and new fundamental control structure of computing, independent of already existing, in production, and arguably better solutions; and independent of analyzing the new problems their silver bullet introduces.
- iainmerrick 8y ago
  “Formal verification” for Rust code -- does that actually exist yet, or do people just hope/assume that the language will be proven consistent and that awesome theorem-checking tools will emerge eventually? I agree that Rust has a great model that seems to lead to very solid code, but “formal verification” is a high bar to clear.

wilun 8y ago

I found it still an interesting read, but I agree with you. The most important problem is: > Then our guarantee is lost: the operations that look like they're inside the with block might actually keep running after the with block ends, and then crash because the file gets closed while they're still using it. And again, you can't tell from local inspection; to know if this is happening you have to go read the source code to all the functions called inside the ... code. But actually you can "tell", or even better have a type system good enough to prevent such mistakes entirely, and the author even knows a bit about Rust, yet fails to state that at least that item is a solved problem there (by using a different and arguably more general approach). Now I don't know enough to decide whether something is missing on the panicking background thread front, but if it does that seems very solvable. I don't buy that spawning threads (or even moral equivalents) and multiprogrammation is in a situation similar to unstructured use of goto anyway. You have tons of problems applicable to one and not the other. And of course resource management is hard to get right with threads. But we have at least a production example of a language that get it right on some points by leveraging more general ideas (and the other difficult points are mostly not addressed by the nursery idea, anyway).