6 ms·

Ok, I'll chime in here since all the comments are missing the point. If you are excited by golang-style in C, then Mr Sustrik's libmill is for you http://libmi

by majke 9y ago

Ok, I'll chime in here since all the comments are missing the point. If you are excited by golang-style in C, then Mr Sustrik's libmill is for you http://libmill.org/tutorial.html http://libmill.org/tutorial.html LibDill is a thin experiment "above" of libmill. The main problem it tries to solve is "cancellations". For the uninitiated, in normal programming languages there is no way to cancel/interrupt a tread/coroutine from outside. Look at the mess in pthread_cancel(3) [1]. In golang as well, there is no way to cancel a coroutine. Why this is important? Well, read Mr Sustrik's notes [2]. But basically - imagine you want to define the time limit for a completion of some task. Or - imagine what should happen if a HTTP request comes, initiates some actions (SQL quieries?) and then disconnects. Normal programming languages have NO way of expressing that - "client went away, stop processing its sql queries". Libdill is an attempt to solve this. I must admit, I'm not fully on board with the "structured concurrency" train of thought [3] , but the whole prospect of defining semantics of killing/cancelling coroutines is absolutely amazing. Also, review the golang "context" mess, which AFAIU tries to work around the same problem. [4] [1] http://man7.org/linux/man-pages/man3/pthread_cancel.3.html http://man7.org/linux/man-pages/man3/pthread_cancel.3.html [2] http://250bpm.com/blog:71 http://250bpm.com/blog:71 [3] http://libdill.org/structured-concurrency.html http://libdill.org/structured-concurrency.html [4] https://golang.org/pkg/context https://golang.org/pkg/context

rumcajz 9y ago
Author here. Thanks for the nice summary. You've expressed it better then I would. I am not 100% sure about the structured concurrency concept myself, but comparing it to structured programming feels reassuring. Basically, a tree structure (whether a syntactic tree or the tree of running coroutines) is probably the most complex way to structuring stuff that's still tractable by a human being. Once you go beyond that, you'll eventually get lost. Thus, tools enforcing a tree-like structure are likely to be helpful to keep the program maintainable.
- igravious 9y ago
  I like the idea of structured tree-like concurrency, fits my brains. About your API though. Why is there a load of inet/ip/socket functions, shouldn't they be in a separate lib? Why is the opposite of the fn int go(expr) the fn int hclose(int) and not the fn int stop(int) or something like it. open/close – go/stop – and so on …
  - rumcajz 9y ago
    The socket functions were once in a separate library but it's just easier to use like this. Linking with 1 library, not having to care about version mismatches etc. My recommendation would be to just not use the socket functions and link with the library stitically. That way the unused code will be discarded. As for hclose() that was just mimicking the POSIX close() function.
    - igravious 9y ago
      Gotcha. Okay.
ewrcoffee 9y ago
Not sure why don't you like go's context model. It works pretty well in our production system. I can see people would like to pass the context implicitly. But I don't have a strong opinion. Explicitness is not that bad.
- fpoling 9y ago
  Go context is a step in the very right direction, but we have not yet the destination. First there is legacy libraries that do not implement it. Second even system-level support misses pieces. For an example, try to cancel a read from a pile using it.
- PuerkitoBio 9y ago
  Go's context is fine to signal cancellation, but it lacks a way to wait for that cancellation to complete gracefully. And the fact that it is also an arbitrary bag of values is concerning. It worries me that it started "polluting" the stdlib's API without solving those issues first. Dave Cheney wrote a good article about it: https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation https://dave.cheney.net/2017/08/20/context-isnt-for-cancella...
- lobster_johnson 9y ago
  Context is problematic because it infects everything. If a function gains context support, every call to it has to be updated, and if you're going to do it right (and not just use context.Background()), every function calling those functions need to be updated, and so on. And the context has to be passed downwards, too. It's turtles all the way up and down. There's been talk of adding context to io.Reader and io.Writer, which would of course affect pretty much every single Go codebase. For backwards compatibility/convenience you therefore see a lot of APIs with complementary functions Foo and FooCtx, one which takes a context and one that doesn't. That's awkward, and even less elegant when interfaces are involved. And the network packages (sockets, etc.), of course, have a parallel, non-complementary mechanism to provide timeouts and cancelation that doesn't use contexts. (The key/value pair context stuff is also unfortunate, in my opinion. It's not typesafe, and results in awkward APIs.) All of this happened because goroutines, as designed, cannot be forcibly terminated. I don't know what the better solution is. I'm not sure why the runtime cannot provoke a panic at the next context switch, but I bet there's a good technical reason. It's unfortunate. But it's not unreasonable that a large part of any codebase needs "contexts", and so if everyone wants it, why not make it mandatory? Make it an implicit argument threaded through every function call, a bit like "this" in languages such as C++ and Java. The compiler could optimize it away from those codepaths that are guaranteed not to need it. Go has a lot of built-in generic magic, adding some global helpers to set a scope's context wouldn't be too bad. The other route is to imitate Erlang's processes, which have their own heap, and so killing a process doesn't leave cruft behind. Given how goroutines are allowed to share memory, that's not likely to happen.
Peaker 9y ago
We use D fibers. When we want to cancel a fiber we throw an exception at it. When it next wakes up, it'll throw (much like sending an exception to a Python generator) -- which will make sure RAII cleanups take place and are not missed.
- aidenn0 9y ago
  Getting that right at the implementation level is quite hard; it also requires support at the OS level that isn't always there (e.g. interrupting a blocking system call).
  - Peaker 9y ago
    Then avoid blocking system calls. When you can't, defer them to a thread that can be ignored.
    - aidenn0 9y ago
      That's a huge footgun in an environment where blocking calls are the default way to get almost anything done, and some calls cannot be made non-blocking.
      - Peaker 9y ago
        True, but we don't have much trouble with it though. We don't directly call syscalls but use our own wrappers that take care of this for us.
        aidenn0 9y ago
        Sorry for the late reply, but I just saw this reply. Can you share who "we" are? I'm always interested in new practical systems for concurrency.

by majke 9y ago

rumcajz 9y ago

Author here. Thanks for the nice summary. You've expressed it better then I would. I am not 100% sure about the structured concurrency concept myself, but comparing it to structured programming feels reassuring. Basically, a tree structure (whether a syntactic tree or the tree of running coroutines) is probably the most complex way to structuring stuff that's still tractable by a human being. Once you go beyond that, you'll eventually get lost. Thus, tools enforcing a tree-like structure are likely to be helpful to keep the program maintainable.

igravious 9y ago
I like the idea of structured tree-like concurrency, fits my brains. About your API though. Why is there a load of inet/ip/socket functions, shouldn't they be in a separate lib? Why is the opposite of the fn int go(expr) the fn int hclose(int) and not the fn int stop(int) or something like it. open/close – go/stop – and so on …
- rumcajz 9y ago
  The socket functions were once in a separate library but it's just easier to use like this. Linking with 1 library, not having to care about version mismatches etc. My recommendation would be to just not use the socket functions and link with the library stitically. That way the unused code will be discarded. As for hclose() that was just mimicking the POSIX close() function.
  - igravious 9y ago
    Gotcha. Okay.

ewrcoffee 9y ago

Not sure why don't you like go's context model. It works pretty well in our production system. I can see people would like to pass the context implicitly. But I don't have a strong opinion. Explicitness is not that bad.

fpoling 9y ago
Go context is a step in the very right direction, but we have not yet the destination. First there is legacy libraries that do not implement it. Second even system-level support misses pieces. For an example, try to cancel a read from a pile using it.
PuerkitoBio 9y ago
Go's context is fine to signal cancellation, but it lacks a way to wait for that cancellation to complete gracefully. And the fact that it is also an arbitrary bag of values is concerning. It worries me that it started "polluting" the stdlib's API without solving those issues first. Dave Cheney wrote a good article about it: https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation https://dave.cheney.net/2017/08/20/context-isnt-for-cancella...
lobster_johnson 9y ago
Context is problematic because it infects everything. If a function gains context support, every call to it has to be updated, and if you're going to do it right (and not just use context.Background()), every function calling those functions need to be updated, and so on. And the context has to be passed downwards, too. It's turtles all the way up and down. There's been talk of adding context to io.Reader and io.Writer, which would of course affect pretty much every single Go codebase. For backwards compatibility/convenience you therefore see a lot of APIs with complementary functions Foo and FooCtx, one which takes a context and one that doesn't. That's awkward, and even less elegant when interfaces are involved. And the network packages (sockets, etc.), of course, have a parallel, non-complementary mechanism to provide timeouts and cancelation that doesn't use contexts. (The key/value pair context stuff is also unfortunate, in my opinion. It's not typesafe, and results in awkward APIs.) All of this happened because goroutines, as designed, cannot be forcibly terminated. I don't know what the better solution is. I'm not sure why the runtime cannot provoke a panic at the next context switch, but I bet there's a good technical reason. It's unfortunate. But it's not unreasonable that a large part of any codebase needs "contexts", and so if everyone wants it, why not make it mandatory? Make it an implicit argument threaded through every function call, a bit like "this" in languages such as C++ and Java. The compiler could optimize it away from those codepaths that are guaranteed not to need it. Go has a lot of built-in generic magic, adding some global helpers to set a scope's context wouldn't be too bad. The other route is to imitate Erlang's processes, which have their own heap, and so killing a process doesn't leave cruft behind. Given how goroutines are allowed to share memory, that's not likely to happen.

Peaker 9y ago

We use D fibers. When we want to cancel a fiber we throw an exception at it. When it next wakes up, it'll throw (much like sending an exception to a Python generator) -- which will make sure RAII cleanups take place and are not missed.

aidenn0 9y ago
Getting that right at the implementation level is quite hard; it also requires support at the OS level that isn't always there (e.g. interrupting a blocking system call).
- Peaker 9y ago
  Then avoid blocking system calls. When you can't, defer them to a thread that can be ignored.
  - aidenn0 9y ago
    That's a huge footgun in an environment where blocking calls are the default way to get almost anything done, and some calls cannot be made non-blocking.
    - Peaker 9y ago
      True, but we don't have much trouble with it though. We don't directly call syscalls but use our own wrappers that take care of this for us.
      - aidenn0 9y ago
        Sorry for the late reply, but I just saw this reply. Can you share who "we" are? I'm always interested in new practical systems for concurrency.