6 ms·

Clojure Async Flow Guide

213 points by simonpure 1y ago

pgorczak 1y ago
This reminded me of Elixir’s GenStage at first glance. Now I wonder how the underlying libs OTP and core.async relate conceptually and implementation wise.
kaliszad 1y ago
Clojure(Script) is a mature, boring in the good sense language and ecosystem. It opened a whole subculture and dialects/ implementations like ClojureCLR, Babashka, nbb, Jank, Janet, Fennel, Joker, Basilisp, Hy, Clojerl and at least a few others I forgot to mention that can perhaps be found on this list: https://github.com/chr15m/awesome-clojure-likes https://github.com/chr15m/awesome-clojure-likes We are building apps for our clients in it, we will also have our own product built with Clojure and ClojureScript soon.
tomconnors 1y ago
I recently used c.a.flow for a program that reads files from s3, massages the data a bit, and writes to a database. The code came out very easy to understand. The challenges were: - getting back pressure right. You need to tune the buffers for the input chans correctly, which takes some thinking, and in my case, some blowing up in prd. - flow monitor (the UI for debugging flows) can't handle large process states, which makes it pretty much useless for my program. - understanding the flow as a whole (rather than specific processes, which are easy to understand in isolation). For example, answering questions like "why hasn't this process received a message for a while?" was tricky.
- puredanger 1y ago
  Thanks for the feedback! Flow monitor does now support filters on the process state (and more on that it is coming to flow itself soon). If you were able to use monitor, it shows the channel buffer states, I guess that was not sufficient to guess why values weren't flowing?
725686 1y ago
Is Clojure still a thing? I sure would hope so, but I haven't seen much of Clojure activity in HN recently.
- ndr 1y ago
  I sense Clojure attracts a bunch of people that prefers building stuff than talking about building stuff. It's bad for marketing, but seems promising for the project longevity. I thought Rich was being a bit pretentious when he compared to violin, but few people ask "are violins still a thing?"
- giancarlostoro 1y ago
  A lot of the Clojure editors I liked that were made specifically for Clojure seemed to have died down too. It's a shame, they were cool and unique. Between Clojure and Racket, those have always been my two favorite Lisp / Scheme languages. I don't do a lot of Lisp but when I do its either in Clojure (thanks to Lein) or in Racket.
  - raspasov 1y ago
    Cursive, Calva, and CIDER are all excellent and very well-supported options. What are you missing specifically?
    - tincholio 1y ago
      Not the parent poster, but I suspect he's thinking of stuff like Lighttable, or Liquid. Editors that were written in Clojure, or specifically for Clojure, and had cool features that were not available elsewhere at the time. (I'm boring myself, and pretty much only ever used Emacs with Cider)
      - giancarlostoro 1y ago
        Correct, there's something I enjoy about editors designed for specific languages in mind, they feel like they're more refined. Like I use the really simple UI that Racket comes with Dr. Racket because it has some unique features, and it just works.
robto 1y ago
I've been meaning to try this out, from my read it's a declarative way to get some structured concurrency. I work in a codebase that heavily uses core.async channels to manage concurrency and you really need to pay close attention to error handling. When you're spawning new threads you need to set up your own custom machinery to re-throw errors on a chans, close chans, and it looks like core.async.flow is a way to do all of this declaratively. Just like `core.async` itself was a frontrunner of Virtual Threads on the JVM, I view `core.async.flow` as the Clojure version of the upcoming [0]Structured Concurrency JEP. I do wonder if it will use that under the hood once it becomes stable, the same way `core.async` is planning to move away from the `go` macro to just have it dispatch a virtual thread. [0]https://openjdk.org/jeps/453 https://openjdk.org/jeps/453
- xmcqdpt2 1y ago
  We've been looking at virtual threads in a project at work and what we found is that it is quite difficult to adapt existing code to run with virtual threads. For example, class initialization pins a thread so any singleton defined in the standard, recommended Java way (using a static inner instance of an inner class) can hang your program forever if the singleton definition suspends. And because they worked really hard on making async colourless, there is no way to know that a method call will suspend. This is a known issue with a workaround if the suspend is due to a network call, https://bugs.openjdk.org/browse/JDK-8355036 https://bugs.openjdk.org/browse/JDK-8355036 which is useful for some applications (web servers). Figuring out that this is why my program was hanging required quite a bit of work too. We are still frustratingly far from the ergonomics of Go concurrency (all threads are virtual threads, hangs automatically panic).
  - puredanger 1y ago
    Clojure's focus on immutable data and pure functions side-step a lot of the trickiest issues with virtual threads. It's often not hard to isolate the I/O parts of your program into flow processes at the edges that can be mapped to the :io pool using virtual threads.
    - 1y ago
user3939382 1y ago
I think LISP is cool and want to use it more but I have 0 appetite to learn the toolchain and debug etc for JVM. You have Racket but Clojure ecosystem is already tiny.
- tombert 1y ago
  Leiningen and deps.edn shield you a bit from the awfulness of Java project management. They feel a lot more like something you'd see in Node.js or something, but it still gets dependencies from Maven Central. Debugging and profiling is still somewhat Java based, and yeah that's can be irritating, but you get used to it. Personally I do think that it's worth it; Clojure is a very pleasant language that has no business being as fast as it is, and core.async is an absolutely lovely concurrency framework if you can convert your logic into messaging, and you have Haskell-style transactional memory for stuff that can't be. So many problems become less irritating in Clojure.
  - dapperdrake 1y ago
    Is clj-boot still a thing or was it ever a thing? It or or was a build tool like Leiningen.
    - tombert 1y ago
      I've never used clj-boot. I've historically mostly used Leiningen but for the last year or so I finally migrated over to deps.edn.
    - Volundr 1y ago
      Boot seems to have pretty much stalled out. I think the builtin Clojure CLI/deps.edn killed off what momentum it did have.
- raspasov 1y ago
  That’s a common misconception. JVM toolchain is way better than the hellscape that are most other language ecosystems. Maven, for example, works reliably and is rock-solid. The only unsolved problem is if you get two libraries/frameworks requesting another dependency, but with different incompatible versions. But I don’t think most other ecosystems solve that painlessly either. Debug specifically is state-of-the-art. Look at YourKit, or any debugger included with common IDEs. None of those tools has a shiny Visual 2025 aesthetic, but again, they work reliably and are going to work the same way a year from now.
Abdii430 1y ago
https://www.facebook.com/profile.php?id=61579582377159 https://www.facebook.com/profile.php?id=61579582377159
deleted 1y ago
[deleted]
yayitswei 1y ago
See also for related ideas: missionary/electric for frontend and rama for backend. I wish for a unified interface combining the best of all three!
kasajian 1y ago
Funny. I was just thinking about dismissing Clojure for a project I'm going to work on because I was concerned about it's lack of ability to work with async calls. I'm too used to how async in JavaScript and C#, and I'm not sure I'd want to work in an environment that doesn't have a simple way to structure async calls. It doesn't necessarily have to be async / await. Just some attention to the issue rather than completely ignoring it.
- mechanicum 1y ago
  I’m curious what led to that conclusion. As far as I remember, making concurrency easier to manage was always presented as one of Clojure’s primary objectives. It’s fundamental to the design e.g. a major motivation for all core data structures being immutable. STM, atoms and agents were there from the beginning. I think futures and promises were added in 1.1. core.async is from 2013. Even popular third-party libraries like promesa and manifold are around 10 years old at this point. I think flow promises to make it easier to orchestrate core.async, specifically, in complex applications, but the essential primitives are far from new and I don’t consider them any harder to use than JavaScript.
- didibus 1y ago
  Async/await for Clojure: https://github.com/xadecimal/async-style https://github.com/xadecimal/async-style
- raspasov 1y ago
  That has always been one of Clojure's main strengths (async & concurrency). With the new JVM VirtualThreads, things are looking better than ever. The transition of core.async specifically to VirtualThreads is still WIP as far as I understand, but with minimal tweaks, 90% of the benefits are already there, even with the current latest version.
  - puredanger 1y ago
    Virtual thread support in core.async is imminent, should land any day now.
nromiun 1y ago
This is cool. So basically you create a group of threads and you can treat them as one unit. Trio does something similar (structured concurrency) with async functions for Python. Are these OS threads or green threads?
- didibus 1y ago
  They can be OS threads or green threads, you can choose.
  - nromiun 1y ago
    Are OS threads still the default and you have to pass a virtual thread executor to use virtual threads?
    - puredanger 1y ago
      When you deploy a flow, you choose the workload type (:compute, :io, or :mixed) and the process will be deployed with the appropriate thread type.
judge123 1y ago
For me, the real 'click' moment with core.async wasn't just about replacing callbacks. It was realizing I could model entire business processes as a system of communicating channels. It completely changed how I think about system design.
- akkad33 1y ago
  Is it like elixir actors
  - lgrapenthin 1y ago
    No. Those have unbounded message queues.
    - akkad33 1y ago
      Thanks. Is that the only difference
      - didibus 1y ago
        It's quite different, the way you model things, it's entirely different. Even though both are able to deliver highly concurrent computation. A process in Elixir has an id, state, and is sent messages to its address. The message are queued inside the process, and handled one at a time. One process can spawn another, and so on. It's more like white collar workers sending emails to each other. In core.async, a Process is anonymous, it doesn't have an id and doesn't have an address. You cannot send messages to it. Instead a process is more like a worker on an assembly line with conveyor belts. What matters are the conveyor belts and what's on them. Where things go from one belt to the next and what happens to the things as they flow through. You can have multiple workers working a belt and if something jams dependent belts stop. The belts are called Channels.
- blackbear_ 1y ago
  Curious if you have found any resources to learn more about this way of thinking?
jibal 1y ago
typos (from a quick surface scan; I'm not familiar with Clojure or this package): "The description arity takes the current state ..." should be "The transition arity takes the current state ..." "excepiton" should be "exception" "Exceptions thrown from the transition or transform arities, the exception ..." should be "If exceptions are thrown from the transition or transform arities, they ..." "provded" should be "provided"
- puredanger 1y ago
  Fixed, thx.
whacked_new 1y ago
so is this something like core's interpretation of things like ztellman's manifold (https://github.com/clj-commons/manifold https://github.com/clj-commons/manifold)? EDIT: gpt says, Choose core.async.flow when you want a declarative, monitorable process graph over channels (pause/resume, error channel, flow monitor). Choose Manifold when you need deferreds/streams as general primitives, precise per-op backpressure, error-aware chaining, or you’re in the Aleph/Netty stack. very abstract but based on this, the answer is no.
stbev 1y ago
This is really interesting, although I still can't get my head around the fact that core.async.flow topologies are immutable. I feel like most problems can't be solved with fixed topologies. I guess one could in theory swap flows the same way values are swapped, but I wonder if this is the way this library is supposed to be used. I also wonder what happens to non-empty channel buffers in this case. I am curious to hear other opinions.
- puredanger 1y ago
  Flow is intended for processes with long-running stable topologies. Rich has been thinking about options to "patch" the running topology but it is quite tricky due to the concurrency issues and I'm not sure that will ever be added. Even though the flow topology is fixed, it's perfectly acceptable for a flow component to use other variable resources and act merely as a coordinator. So you could for example have a process that send data out to an external dynamic thread pool and gets callbacks via a channel.
vim-guru 1y ago
So, kind of the opposite of https://github.com/lovrosdu/klor https://github.com/lovrosdu/klor but with the same goal
- raspasov 1y ago
  I am not sure if this is “opposite” but on the surface looks very interesting!

213 points by simonpure 1y ago

This reminded me of Elixir’s GenStage at first glance. Now I wonder how the underlying libs OTP and core.async relate conceptually and implementation wise.

kaliszad 1y ago

Clojure(Script) is a mature, boring in the good sense language and ecosystem. It opened a whole subculture and dialects/ implementations like ClojureCLR, Babashka, nbb, Jank, Janet, Fennel, Joker, Basilisp, Hy, Clojerl and at least a few others I forgot to mention that can perhaps be found on this list: https://github.com/chr15m/awesome-clojure-likes https://github.com/chr15m/awesome-clojure-likes We are building apps for our clients in it, we will also have our own product built with Clojure and ClojureScript soon.

tomconnors 1y ago

I recently used c.a.flow for a program that reads files from s3, massages the data a bit, and writes to a database. The code came out very easy to understand. The challenges were: - getting back pressure right. You need to tune the buffers for the input chans correctly, which takes some thinking, and in my case, some blowing up in prd. - flow monitor (the UI for debugging flows) can't handle large process states, which makes it pretty much useless for my program. - understanding the flow as a whole (rather than specific processes, which are easy to understand in isolation). For example, answering questions like "why hasn't this process received a message for a while?" was tricky.

puredanger 1y ago
Thanks for the feedback! Flow monitor does now support filters on the process state (and more on that it is coming to flow itself soon). If you were able to use monitor, it shows the channel buffer states, I guess that was not sufficient to guess why values weren't flowing?

725686 1y ago

Is Clojure still a thing? I sure would hope so, but I haven't seen much of Clojure activity in HN recently.

ndr 1y ago
I sense Clojure attracts a bunch of people that prefers building stuff than talking about building stuff. It's bad for marketing, but seems promising for the project longevity. I thought Rich was being a bit pretentious when he compared to violin, but few people ask "are violins still a thing?"
giancarlostoro 1y ago
A lot of the Clojure editors I liked that were made specifically for Clojure seemed to have died down too. It's a shame, they were cool and unique. Between Clojure and Racket, those have always been my two favorite Lisp / Scheme languages. I don't do a lot of Lisp but when I do its either in Clojure (thanks to Lein) or in Racket.
- raspasov 1y ago
  Cursive, Calva, and CIDER are all excellent and very well-supported options. What are you missing specifically?
  - tincholio 1y ago
    Not the parent poster, but I suspect he's thinking of stuff like Lighttable, or Liquid. Editors that were written in Clojure, or specifically for Clojure, and had cool features that were not available elsewhere at the time. (I'm boring myself, and pretty much only ever used Emacs with Cider)
    - giancarlostoro 1y ago
      Correct, there's something I enjoy about editors designed for specific languages in mind, they feel like they're more refined. Like I use the really simple UI that Racket comes with Dr. Racket because it has some unique features, and it just works.

robto 1y ago

I've been meaning to try this out, from my read it's a declarative way to get some structured concurrency. I work in a codebase that heavily uses core.async channels to manage concurrency and you really need to pay close attention to error handling. When you're spawning new threads you need to set up your own custom machinery to re-throw errors on a chans, close chans, and it looks like core.async.flow is a way to do all of this declaratively. Just like `core.async` itself was a frontrunner of Virtual Threads on the JVM, I view `core.async.flow` as the Clojure version of the upcoming [0]Structured Concurrency JEP. I do wonder if it will use that under the hood once it becomes stable, the same way `core.async` is planning to move away from the `go` macro to just have it dispatch a virtual thread. [0]https://openjdk.org/jeps/453 https://openjdk.org/jeps/453

xmcqdpt2 1y ago
We've been looking at virtual threads in a project at work and what we found is that it is quite difficult to adapt existing code to run with virtual threads. For example, class initialization pins a thread so any singleton defined in the standard, recommended Java way (using a static inner instance of an inner class) can hang your program forever if the singleton definition suspends. And because they worked really hard on making async colourless, there is no way to know that a method call will suspend. This is a known issue with a workaround if the suspend is due to a network call, https://bugs.openjdk.org/browse/JDK-8355036 https://bugs.openjdk.org/browse/JDK-8355036 which is useful for some applications (web servers). Figuring out that this is why my program was hanging required quite a bit of work too. We are still frustratingly far from the ergonomics of Go concurrency (all threads are virtual threads, hangs automatically panic).
- puredanger 1y ago
  Clojure's focus on immutable data and pure functions side-step a lot of the trickiest issues with virtual threads. It's often not hard to isolate the I/O parts of your program into flow processes at the edges that can be mapped to the :io pool using virtual threads.
  - 1y ago

user3939382 1y ago

I think LISP is cool and want to use it more but I have 0 appetite to learn the toolchain and debug etc for JVM. You have Racket but Clojure ecosystem is already tiny.

tombert 1y ago
Leiningen and deps.edn shield you a bit from the awfulness of Java project management. They feel a lot more like something you'd see in Node.js or something, but it still gets dependencies from Maven Central. Debugging and profiling is still somewhat Java based, and yeah that's can be irritating, but you get used to it. Personally I do think that it's worth it; Clojure is a very pleasant language that has no business being as fast as it is, and core.async is an absolutely lovely concurrency framework if you can convert your logic into messaging, and you have Haskell-style transactional memory for stuff that can't be. So many problems become less irritating in Clojure.
- dapperdrake 1y ago
  Is clj-boot still a thing or was it ever a thing? It or or was a build tool like Leiningen.
  - tombert 1y ago
    I've never used clj-boot. I've historically mostly used Leiningen but for the last year or so I finally migrated over to deps.edn.
  - Volundr 1y ago
    Boot seems to have pretty much stalled out. I think the builtin Clojure CLI/deps.edn killed off what momentum it did have.
raspasov 1y ago
That’s a common misconception. JVM toolchain is way better than the hellscape that are most other language ecosystems. Maven, for example, works reliably and is rock-solid. The only unsolved problem is if you get two libraries/frameworks requesting another dependency, but with different incompatible versions. But I don’t think most other ecosystems solve that painlessly either. Debug specifically is state-of-the-art. Look at YourKit, or any debugger included with common IDEs. None of those tools has a shiny Visual 2025 aesthetic, but again, they work reliably and are going to work the same way a year from now.

Abdii430 1y ago

https://www.facebook.com/profile.php?id=61579582377159 https://www.facebook.com/profile.php?id=61579582377159

deleted 1y ago

[deleted]

yayitswei 1y ago

See also for related ideas: missionary/electric for frontend and rama for backend. I wish for a unified interface combining the best of all three!

kasajian 1y ago

Funny. I was just thinking about dismissing Clojure for a project I'm going to work on because I was concerned about it's lack of ability to work with async calls. I'm too used to how async in JavaScript and C#, and I'm not sure I'd want to work in an environment that doesn't have a simple way to structure async calls. It doesn't necessarily have to be async / await. Just some attention to the issue rather than completely ignoring it.

mechanicum 1y ago
I’m curious what led to that conclusion. As far as I remember, making concurrency easier to manage was always presented as one of Clojure’s primary objectives. It’s fundamental to the design e.g. a major motivation for all core data structures being immutable. STM, atoms and agents were there from the beginning. I think futures and promises were added in 1.1. core.async is from 2013. Even popular third-party libraries like promesa and manifold are around 10 years old at this point. I think flow promises to make it easier to orchestrate core.async, specifically, in complex applications, but the essential primitives are far from new and I don’t consider them any harder to use than JavaScript.
didibus 1y ago
Async/await for Clojure: https://github.com/xadecimal/async-style https://github.com/xadecimal/async-style
raspasov 1y ago
That has always been one of Clojure's main strengths (async & concurrency). With the new JVM VirtualThreads, things are looking better than ever. The transition of core.async specifically to VirtualThreads is still WIP as far as I understand, but with minimal tweaks, 90% of the benefits are already there, even with the current latest version.
- puredanger 1y ago
  Virtual thread support in core.async is imminent, should land any day now.

nromiun 1y ago

This is cool. So basically you create a group of threads and you can treat them as one unit. Trio does something similar (structured concurrency) with async functions for Python. Are these OS threads or green threads?

didibus 1y ago
They can be OS threads or green threads, you can choose.
- nromiun 1y ago
  Are OS threads still the default and you have to pass a virtual thread executor to use virtual threads?
  - puredanger 1y ago
    When you deploy a flow, you choose the workload type (:compute, :io, or :mixed) and the process will be deployed with the appropriate thread type.

judge123 1y ago

For me, the real 'click' moment with core.async wasn't just about replacing callbacks. It was realizing I could model entire business processes as a system of communicating channels. It completely changed how I think about system design.

akkad33 1y ago
Is it like elixir actors
- lgrapenthin 1y ago
  No. Those have unbounded message queues.
  - akkad33 1y ago
    Thanks. Is that the only difference
    - didibus 1y ago
      It's quite different, the way you model things, it's entirely different. Even though both are able to deliver highly concurrent computation. A process in Elixir has an id, state, and is sent messages to its address. The message are queued inside the process, and handled one at a time. One process can spawn another, and so on. It's more like white collar workers sending emails to each other. In core.async, a Process is anonymous, it doesn't have an id and doesn't have an address. You cannot send messages to it. Instead a process is more like a worker on an assembly line with conveyor belts. What matters are the conveyor belts and what's on them. Where things go from one belt to the next and what happens to the things as they flow through. You can have multiple workers working a belt and if something jams dependent belts stop. The belts are called Channels.
blackbear_ 1y ago
Curious if you have found any resources to learn more about this way of thinking?

jibal 1y ago

typos (from a quick surface scan; I'm not familiar with Clojure or this package): "The description arity takes the current state ..." should be "The transition arity takes the current state ..." "excepiton" should be "exception" "Exceptions thrown from the transition or transform arities, the exception ..." should be "If exceptions are thrown from the transition or transform arities, they ..." "provded" should be "provided"

puredanger 1y ago
Fixed, thx.

whacked_new 1y ago

so is this something like core's interpretation of things like ztellman's manifold (https://github.com/clj-commons/manifold https://github.com/clj-commons/manifold)? EDIT: gpt says, Choose core.async.flow when you want a declarative, monitorable process graph over channels (pause/resume, error channel, flow monitor). Choose Manifold when you need deferreds/streams as general primitives, precise per-op backpressure, error-aware chaining, or you’re in the Aleph/Netty stack. very abstract but based on this, the answer is no.

stbev 1y ago

This is really interesting, although I still can't get my head around the fact that core.async.flow topologies are immutable. I feel like most problems can't be solved with fixed topologies. I guess one could in theory swap flows the same way values are swapped, but I wonder if this is the way this library is supposed to be used. I also wonder what happens to non-empty channel buffers in this case. I am curious to hear other opinions.

puredanger 1y ago
Flow is intended for processes with long-running stable topologies. Rich has been thinking about options to "patch" the running topology but it is quite tricky due to the concurrency issues and I'm not sure that will ever be added. Even though the flow topology is fixed, it's perfectly acceptable for a flow component to use other variable resources and act merely as a coordinator. So you could for example have a process that send data out to an external dynamic thread pool and gets callbacks via a channel.

vim-guru 1y ago

So, kind of the opposite of https://github.com/lovrosdu/klor https://github.com/lovrosdu/klor but with the same goal

raspasov 1y ago
I am not sure if this is “opposite” but on the surface looks very interesting!