6 ms·

Clojure Concurrency Tutorial

222 points by maastaar 8y ago

tombert 8y ago
I get to do Clojure about ~60% of the time at work now, and it never ceases to amaze me how much easier it is to deal with concurrency than in vanilla Java. Futures and Promises and core.async don't allow you to totally avoid planning out your project (I've been bitten slightly by core.async's `go` blocks behaving differently than Go's goroutines), but they are a godsend compared to dealing with manual mutexes and semaphores. I still need to play with manifold; I hear that it helps deal with the IO-heavy stuff that core.async chokes on.
- quadcore 8y ago
  What's the difference between core.async go blocks and goroutine please, I thought they would be the same to a reasonable extent?
  - tombert 8y ago
    To a reasonable extent, yes, they're the same. The biggest issue is that goroutines are (more or less) fully preemptive (for all intents and purpose) with their thread pooling and go blocks in core.async are not. Basically, in Go, you can have as many blocking IO thing taking millions of seconds to run, but it won't block the thread pool, and your other goroutines will run fine. Since core.async go blocks aren't fully preemptive, long-running things can eat up the whole thread pool. core.async allocates "number of cores + 2" number of "real" threads to do its work. For example, say I have a program that downloads hundreds of giant files running on an 8-core server. The naive implementation would have a goroutine/go block for each concurrent download. In Go, this would work more or less ok (assuming you have plenty of memory), but in Clojure it would block all the other go blocks after 10 until the download is done. Normally, it's not an issue, but it's a small gotcha if you're not paying enough attention.
    - quadcore 8y ago
      I'm a bit surprised if I understand you correctly. I though a goroutine would run and hold the thread until any kind of sleep. edit: Oh but there might be no "core.async sleep" in most of clojure libraries like IO/net stuff, which would explain why async go blocks hold the thread more often than they should, whereas Go has "goroutines sleeps" everywhere.
      - setr 8y ago
        As I recall it, goroutines are in fact cooperative with implicit yield points. The implicit behavior is what I believe go means by “for all intents and purposes” Particularly, I believe all function calls are implicitly yield, which means you can usually pretend that its preemptive through normal coding practices; but you can still never yield through a while(1){} (i think there was a proposal to add implicit yield on loops? Not sure if it went through) I would assume core.async does not have many, if any, implicit yields, leading to the different behavior. But technically, both are cooperative, go is just more convenient about it
        Blackthorn 8y ago
        The real magic is where you can call blocking functions from goroutines and have it Just Work!
        quadcore 8y ago
        The implicit yield points you're talking about is sleep or wait (noop in other terms). It's also not exactly a yield, it calls the scheduler to tell it to schedule (meaning the scheduler could decide to run again the exact same goroutine that noop-ed). Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.
        setr 8y ago
        >The implicit yield points you're talking about is sleep or wait (noop in other terms) Isn’t sleep() usually defined as not giving up control? It’s semantically a busyspin for n seconds. That is, it’s explicitly not a yield point; it's a blocking call, and it doesn’t give control back to scheduler. And isn't wait() usually defined as, semantically, a non-blocking spinloop with a condition (hence notify() or poll())? Which makes sleep(), wait(), yield() and noops very different. I suppose in go, if yield() is implicit on any function call, then sleep() will immediately call yield() before actually sleeping (but once started, it’ll hold control until finished) I'm pretty sure the implicit yield points are actually yield(), though I haven't verified; it's the only one of the sleep/wait/yield semantics that would be sensible to be implicit as far as I can tell. >It's also not exactly a yield, it calls the scheduler to tell it to schedule As far as I was aware, thats exactly what yield means; it yields execution control back to the scheduler. I suppose in eg, a generator, it yields control back to the caller, but afaik thats fundamentally the same meaning (the caller is the scheduler). Can you expand? >Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads. Sure, but is that relevant? I assume async.core also trivially supports thread pooling, and is presumably equivalent to go’s pool. The only real difference is go implicitly instiates the pool, while core.async presumably makes it an explicit function to call
  - Blackthorn 8y ago
    core.async is a macro. Call an already compiled function under a library that blocks and the whole system will come to a screeching halt. Goroutines don't have that limitation.
    - jwr 8y ago
      Please see my other comment about using the same channels and calling I/O-heavy (blocking) functions from a `(thread)` instead of from a go block. You indeed shouldn't call blocking functions from go blocks, but core.async gives you the option of freely mixing separate threads and go blocks.
      - Blackthorn 8y ago
        Having to use full threads in some cases and core.async in others pretty much defeats the purpose. Core.async is just nowhere near as good as goroutines. It'll continue to be like that until Project Loom finally has a release.
        tombert 8y ago
        Sorry, what is Project Loom? I'm not overly familiar with it and a quick search for `clojure project loom` seems to indicate that it's a graphing library.
        lackbeard 8y ago
        https://wiki.openjdk.java.net/display/loom/Main https://wiki.openjdk.java.net/display/loom/Main
        jwr 7y ago
        I think you misunderstood what I wrote. Core.async lets you freely mix both. You use the same channels, the only difference being that your coroutine blocks will be wrapped in `go` and use `<!` (parks coroutine if it needs to block), while your full threads with be wrapped in `thread` and use `<!!` (blocks). It makes the tradeoffs explicit (the tradeoff always exists) and isn't a problem at all.
- devgoth 8y ago
  I was reading somewhere that the core.async lib has not been updated in awhile...is this correct? (generally curious if the lib is actively updated)
  - didibus 8y ago
    It is maintained, but not actively. Like the last year there was two issues fixed in it. There isn't someone actively trying to enhance it or add more features to it. So my understanding is only major issues gets addressed.
    - siscia 8y ago
      Which is a quite peculiar and reccurrent pattern in closure. After a while libraries and tools stabilize and the community just maintains it.
      - jakeinspace 8y ago
        Most Clojure libraries I've worked with, even the "larger" ones, are fairly small and concise by modern standards. I wonder if this has something to do with the lack of activity (in addition to being a small community). It feels a bit like the Unix philosophy - lots of small/medium libraries handling discrete problems, rather than many competing mega-frameworks. To me, this feels like a silver lining that comes from the limitations of a smaller dev community.
        tombert 8y ago
        This is why I love functional programming as a whole. I feel like my code is really terse, without little/no sacrifices in expressivity, and usually no substantial sacrifice in performance.
        deleted 8y ago
        [deleted]
      - 8y ago
- kitd 8y ago
  I still do quite a bit of Java work. Nowadays, most high-concurrency Java apps use async tools like Vert.x [1], not explicit mutexes & semaphores. In that respect, modern Java is not that dissimilar from core.async, etc. [1] - https://vertx.io/ https://vertx.io/
  - tombert 8y ago
    That's a fair point; I admittedly haven't touched Java concurrency in quite awhile, and I actually haven't touched Vert.x since they had direct Clojure bindings :).
lincpa 8y ago
Data flow programming is a natural Concurrency programming. In addition, Clojure's STM mechanism comes from RMDB's MVCC. It is the easiest, most convenient, and most appropriate to support data flow programming. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow

222 points by maastaar 8y ago

I get to do Clojure about ~60% of the time at work now, and it never ceases to amaze me how much easier it is to deal with concurrency than in vanilla Java. Futures and Promises and core.async don't allow you to totally avoid planning out your project (I've been bitten slightly by core.async's `go` blocks behaving differently than Go's goroutines), but they are a godsend compared to dealing with manual mutexes and semaphores. I still need to play with manifold; I hear that it helps deal with the IO-heavy stuff that core.async chokes on.

quadcore 8y ago
What's the difference between core.async go blocks and goroutine please, I thought they would be the same to a reasonable extent?
- tombert 8y ago
  To a reasonable extent, yes, they're the same. The biggest issue is that goroutines are (more or less) fully preemptive (for all intents and purpose) with their thread pooling and go blocks in core.async are not. Basically, in Go, you can have as many blocking IO thing taking millions of seconds to run, but it won't block the thread pool, and your other goroutines will run fine. Since core.async go blocks aren't fully preemptive, long-running things can eat up the whole thread pool. core.async allocates "number of cores + 2" number of "real" threads to do its work. For example, say I have a program that downloads hundreds of giant files running on an 8-core server. The naive implementation would have a goroutine/go block for each concurrent download. In Go, this would work more or less ok (assuming you have plenty of memory), but in Clojure it would block all the other go blocks after 10 until the download is done. Normally, it's not an issue, but it's a small gotcha if you're not paying enough attention.
  - quadcore 8y ago
    I'm a bit surprised if I understand you correctly. I though a goroutine would run and hold the thread until any kind of sleep. edit: Oh but there might be no "core.async sleep" in most of clojure libraries like IO/net stuff, which would explain why async go blocks hold the thread more often than they should, whereas Go has "goroutines sleeps" everywhere.
    - setr 8y ago
      As I recall it, goroutines are in fact cooperative with implicit yield points. The implicit behavior is what I believe go means by “for all intents and purposes” Particularly, I believe all function calls are implicitly yield, which means you can usually pretend that its preemptive through normal coding practices; but you can still never yield through a while(1){} (i think there was a proposal to add implicit yield on loops? Not sure if it went through) I would assume core.async does not have many, if any, implicit yields, leading to the different behavior. But technically, both are cooperative, go is just more convenient about it
      - Blackthorn 8y ago
        The real magic is where you can call blocking functions from goroutines and have it Just Work!
      - quadcore 8y ago
        The implicit yield points you're talking about is sleep or wait (noop in other terms). It's also not exactly a yield, it calls the scheduler to tell it to schedule (meaning the scheduler could decide to run again the exact same goroutine that noop-ed). Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.
        setr 8y ago
        >The implicit yield points you're talking about is sleep or wait (noop in other terms) Isn’t sleep() usually defined as not giving up control? It’s semantically a busyspin for n seconds. That is, it’s explicitly not a yield point; it's a blocking call, and it doesn’t give control back to scheduler. And isn't wait() usually defined as, semantically, a non-blocking spinloop with a condition (hence notify() or poll())? Which makes sleep(), wait(), yield() and noops very different. I suppose in go, if yield() is implicit on any function call, then sleep() will immediately call yield() before actually sleeping (but once started, it’ll hold control until finished) I'm pretty sure the implicit yield points are actually yield(), though I haven't verified; it's the only one of the sleep/wait/yield semantics that would be sensible to be implicit as far as I can tell. >It's also not exactly a yield, it calls the scheduler to tell it to schedule As far as I was aware, thats exactly what yield means; it yields execution control back to the scheduler. I suppose in eg, a generator, it yields control back to the caller, but afaik thats fundamentally the same meaning (the caller is the scheduler). Can you expand? >Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads. Sure, but is that relevant? I assume async.core also trivially supports thread pooling, and is presumably equivalent to go’s pool. The only real difference is go implicitly instiates the pool, while core.async presumably makes it an explicit function to call
- Blackthorn 8y ago
  core.async is a macro. Call an already compiled function under a library that blocks and the whole system will come to a screeching halt. Goroutines don't have that limitation.
  - jwr 8y ago
    Please see my other comment about using the same channels and calling I/O-heavy (blocking) functions from a `(thread)` instead of from a go block. You indeed shouldn't call blocking functions from go blocks, but core.async gives you the option of freely mixing separate threads and go blocks.
    - Blackthorn 8y ago
      Having to use full threads in some cases and core.async in others pretty much defeats the purpose. Core.async is just nowhere near as good as goroutines. It'll continue to be like that until Project Loom finally has a release.
      - tombert 8y ago
        Sorry, what is Project Loom? I'm not overly familiar with it and a quick search for `clojure project loom` seems to indicate that it's a graphing library.
        lackbeard 8y ago
        https://wiki.openjdk.java.net/display/loom/Main https://wiki.openjdk.java.net/display/loom/Main
      - jwr 7y ago
        I think you misunderstood what I wrote. Core.async lets you freely mix both. You use the same channels, the only difference being that your coroutine blocks will be wrapped in `go` and use `<!` (parks coroutine if it needs to block), while your full threads with be wrapped in `thread` and use `<!!` (blocks). It makes the tradeoffs explicit (the tradeoff always exists) and isn't a problem at all.
devgoth 8y ago
I was reading somewhere that the core.async lib has not been updated in awhile...is this correct? (generally curious if the lib is actively updated)
- didibus 8y ago
  It is maintained, but not actively. Like the last year there was two issues fixed in it. There isn't someone actively trying to enhance it or add more features to it. So my understanding is only major issues gets addressed.
  - siscia 8y ago
    Which is a quite peculiar and reccurrent pattern in closure. After a while libraries and tools stabilize and the community just maintains it.
    - jakeinspace 8y ago
      Most Clojure libraries I've worked with, even the "larger" ones, are fairly small and concise by modern standards. I wonder if this has something to do with the lack of activity (in addition to being a small community). It feels a bit like the Unix philosophy - lots of small/medium libraries handling discrete problems, rather than many competing mega-frameworks. To me, this feels like a silver lining that comes from the limitations of a smaller dev community.
      - tombert 8y ago
        This is why I love functional programming as a whole. I feel like my code is really terse, without little/no sacrifices in expressivity, and usually no substantial sacrifice in performance.
        deleted 8y ago
        [deleted]
    - 8y ago
kitd 8y ago
I still do quite a bit of Java work. Nowadays, most high-concurrency Java apps use async tools like Vert.x [1], not explicit mutexes & semaphores. In that respect, modern Java is not that dissimilar from core.async, etc. [1] - https://vertx.io/ https://vertx.io/
- tombert 8y ago
  That's a fair point; I admittedly haven't touched Java concurrency in quite awhile, and I actually haven't touched Vert.x since they had direct Clojure bindings :).

lincpa 8y ago

Data flow programming is a natural Concurrency programming. In addition, Clojure's STM mechanism comes from RMDB's MVCC. It is the easiest, most convenient, and most appropriate to support data flow programming. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow