9 ms·

My mental model of Clojure transducers

• deleted 3y ago
  [deleted]
• rowbin 3y ago
  So like partially applied functions in Haskell?
  • lgrapenthin 3y ago
    Not even by far
  • mrkeen 3y ago
    Probably just plain old functions & laziness. If you want to interleave IO into it then probably a library like conduit.
    • adityaathalye 3y ago
      Plain old functions & eager evaluation & a bit more awesome sauce. Given transducers, we can compose mutually independent parts at will: - Data source (sequence, stream, channel, socket etc.) - Data sink (sequence, stream, channel, socket etc.) - Data transformer (function of any value -> any other value) - Data transformation process (mapping, filtering, reducing etc.) - Some process control (we can transduce finite data (of course) as well as streams, and also have optional early termination in either case. I'm not sure about first-class support for other methods like backpressure.) e.g. read numbers off a Kafka topic, FizzBuzz them, and send them to another Kafka topic, OR slurp numbers from file on disk, FizzBuzz them, and push into an in-memory queue. But each time, you don't have to rewrite your core fizzbuzz function, nor your `(map fizzbuzz)` definition. cf. https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojure/index.html#transducery-buzz https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojur...
      • throwaway858 3y ago
        I'm not sure why the parent was downvoted, this sounds exactly like the Haskell conduit library (or indeed plain laziness if you don't need IO).
        • waffletower 3y ago
          I can't downvote, but might have as the first sentence is an over-simplication and misunderstanding -- particularly as laziness for collections has always been available in clojure.core. Clojure transducers offer an optimization orthogonal to collections best summed above with: "transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections". Yes, transducers can be viewed as somewhat of an analog to the Haskell conduit library (as discussed here several years ago: https://hypirion.com/musings/haskell-transducers https://hypirion.com/musings/haskell-transducers). However, I think the detractors coming from strongly typed languages are decidedly missing much of the generalization of the transducer model, particularly those conflating transducers exclusively with streams.
          • throwaway858 3y ago
            Thanks for this link. It seems to confirm things: "aren’t Conduits and Transducers then equivalent (isomorphic)? I am pretty sure they are." I view this as a good sign. When two independent parties arrive at the same design it is usually an indication that they have discovered a universal and principled solution. I consider the "conduit" library to be one of Haskell's "killer features", and sorely miss having something like it when working in other languages. Maybe when Haskellers dismiss clojure transducers as being "just like conduit" it comes from a place of jealousy? I've seen several articles and discussions over the years of clojure transducers that take place outside of clojure communities and are aimed at the wider programming public, praising the benefits of it. But I've never seen conduit discussed outside of Haskell communities.
  • bmacho 3y ago
    You probably are thinking of normal functions, and not partially applied ones (which are also just normal functions that we get a special way totally unrelated here). Also I don't think they can reproduce Blammo! Our gnome is now packaging together incoming items into bundles of three, caching them in the interim while the bundle is not complete yet. But if we close the input prematurely, it will acknowledge and produce the incomplete bundle: (>!! b 4) (>!! b 5) (close! b) ; Value: [4 5]
  • crdrost 3y ago
    It's not, this is that Lisp thing where you can check the length of your argument list and do something completely different when you don't get enough arguments. In this case `(map f)` notices that it was told to map a function but not told what to map it over, and so it decides to give you a new function. If this were a partially applied function the signature would be `[x] -> [y]` where `f: x -> y` would pick out the specifics. But this is actually a totally different signature, isomorphic to `x -> [y]`, the signature of generators. Specifically `(map f)` generates what in Haskell would be `\x -> [f x]`. However the type is not quite that straightforward for historical and compositional reasons; it is actually ∀z. (y -> z -> z) -> x -> z -> z With the implementation being here \handle x -> handle (f x) This is a sort of enhanced map that can do filtering because it uses concatMap (also known as >>=, “bind in the list monad”) to combine. So to `(filter pred)` you would have the isomorphic versions, \x -> if pred x then [x] else [] \handle x rest -> if pred x then handle x rest else rest This also leads to an important nitpick for the article in question, a strictly better mental model of a transducer is not that it maps conveyor belts to conveyor belts, since that has more power than transducers do. (For instance, reverse is not a transducer.) But rather that it maps individual items on a conveyor belt, to their own conveyor belts on the first conveyor belt, then mashes them all together into one effective conveyor belt. So filter will either map an object to a singleton conveyor belt containing that thing, or an empty conveyor belt. You can implement `dupIf pred x = if pred x then [ x, x] else [x]` as a transducer too, `handle x (handle x rest)`. Conveyor belt that either has one or two elements on it. You can potentially put an infinite conveyor belt inside your conveyor belts and make a chunk of the input unreachable, although Clojure is strict so I have the feeling this will just run out of memory?
    • rowbin 3y ago
      Thanks, that helped
  • slowmovintarget 3y ago
    Transducers are functions that return transformed reducing functions.
  • lmm 3y ago
    They're like iteratees, but with awkward edge cases (particularly around error handling). If you've used Conduit you'll have already had the positive experiences other comments are talking about - realising how powerful and general-purpose the abstraction is and using it for everything.
• mcbrit 3y ago
  What is generally missing from this category of article is a motivating statement, eg here is a problem that is easier or at least different (transformed into a different category of problem) given this idea. Up top. When I don’t see this up top or scanning forward I assume the article assumes knowledge that I do not have, and I bounce.
  • adityaathalye 3y ago
    True, though I think this is part personal note, and part intended as additional material for people already trying to apply transducers effectively. The author suggests this by casting it as "my" mental model, in a work context. Maybe this variant of explanation will suit your context better: https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojure/index.html#transducery-buzz https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojur... I wrote the post as a way to explore Clojure's standard library using FizzBuzz as a device.
    • mcbrit 3y ago
      Most folks on HN who are interested in PLs, including me, are familiar with transducers at a high level. Composable, performant, yada yada ya. What we (or maybe just I) do not have is a nontrivial example of advantage. Your comment sharpened that for me. I don't want FizzBuzz; I want someone taking a reasonable toy problem, such as a trad+photon+quadtree raytracer and demonstrating advantage by applying the concept.
  • casion 3y ago
    To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. So rather than processing the collection, passing it to the next function that processes the collection, passing it to the next... etc.. consuming all the CPU and memory that involves, you can define steps that are applied for each item in the collection thereby having the iteration through the collection happen once. These steps (transducers) are also composable and reusable. I suspect you know this, consider this a basic explanation for other people reading.
• waffletower 3y ago
  Great that the author mentioned Christophe Grand’s xforms library. It has transducers (and reducing functions) useful for hashmap processing such as x/by-key and x/reduce. Very useful for utilizing the transducer paradigm with a wider array of data types and problem spaces.
• jwr 3y ago
  Transducers are an under-appreciated feature in Clojure. They are incredibly useful, allow for composable and reusable code, and come with really nice performance benefits as well (by not creating intermediate collections). Once you get used to them, they become a very natural tool — in my case, almost every time I do something to a sequence, I'll start with `into`. Even if I'm just applying a single transformation, this lets me quickly change the resulting collection and easily add more transformations if needed. On a higher level, I found myself thinking about business logic (model) in terms of transducer pipelines and ended up with a number of reusable transformations, and clearly specified pipeline logic. One gripe I have with transducers is that writing stateful transducers is hard. As in, well, really hard (hope you remembered to flush using `unreduced` in your single-arity version!). I still write them sometimes, but it's never a walk in the park (it does provide satisfaction when done, though). But I guess that's what you need to go through in order to get good performance.
  • geokon 3y ago
    joinr pointed out this lib to me and it seems to make using them more ergonomic (if you're into pipelining) https://github.com/johnmn3/injest https://github.com/johnmn3/injest However writing your own indeed doesn't sound fun :))
    • jwr 3y ago
      Hmm. I don't see much value in the new syntax: the current syntax works for me just fine.
      • geokon 3y ago
        I use pipelining everywhere. Going from piping through map/reduce/filter/etc back to inside out Lisp style calls felt like a huge ergonomic step back. Maybe it's a matter of familiarity but it's hard for me to visually scan that kind of code
• bjoli 3y ago
  I wrote SRFI-171 for scheme. Ask me anything. I have had many people tell me that the SRFI document [0] made transducers click for them, which is always a nice thing to hear. 0: https://srfi.schemers.org/srfi-171/srfi-171.html https://srfi.schemers.org/srfi-171/srfi-171.html
  • mcbrit 3y ago
    OK. It's a better map that can keep some state. (I scanned not read your doc and that was my takeaway; I have not thought about what I can do with state yet, particularly since I had never associated transducers with being able to keep state) Can you give me an example of a classic problem (since we're talking about transformations, raytracers and compilers come to mind) where if you involve a transducer vs a map you get an interesting difference? Edit: 'How state is kept is not specified': I assume that there are limitations to state keeping, particularly with the composability and performance pillars of transducers, but I'm just having a really hard time synthesizing everything.
    • bjoli 3y ago
      You could keep the state using the state monad, or by letting every reducer keep a transparent state in a linked list that whoever is pushing values through it has to handle. The reference implementation keeps it hidden using closures. This is mostly an API thing. In clojure you can pass a transducer when you create a channel. That way you can make a channel do just about anything. Send data in chunks of N. Filter Odd numbers. Or just do arbitrary transformations. It is a protocol for composable transformations of data being passed in one direction. It is not fancy. Not really hard to understand. A generalization of map, filter, and friends.
    • bjoli 3y ago
      Regarding state: you can make thread safe transducers. The current SRFI 171 reference implementation is NOT thread safe. You can create a transducer and use it across different threads no problem. But you cannot start a transducer and use the returned reducer in different threads. It uses hidden mutable state.
  • 3y ago
• yakubin 3y ago
  How are transducers different than iterators in C++ or Rust or streams in Java?
  • nextaccountic 3y ago
    My reading is that a transducer is a map, that is, something that transforms iterators Like something that receives myiterator and returns myiterator.map(|x| x + 2) in rust But other than focusing on the map rather than on the iterator, I think it's the same thing
    • fiddlerwoaroof 3y ago
      The power it has on top of map is it can skip items and change the shape of the container: that is, it’s a decomposition of a fold and not just a composable map. So, you can consume a vector and produce a subset of the vector as a map. Finally, transducers separate the element-wise processing from both the construction of the final sequence and the details of iterating over the input.
      • nextaccountic 3y ago
        A map that can skip itens is normally called a filter_map But can it produce more than one item as well? So it's like, for each item of the iterator/stream/sequence, it can produce 0, 1, or any number of items? Then it's like Haskell's bind (the >>= of the Monad instance) for lists, or Rust's flat_map for iterators, and flatMap in javascript as well
        • fiddlerwoaroof 3y ago
          It’s more than bind, because a monad requires that the input monad be the same as the output one (if you consume a list, you can only produce a list). Clojure’s Transducers are as powerful as foldr because the output type can be different. E.g.: (into {} (map (fn [k] [(/ k 2) k])) [1 2 3 4]) ;; {1 2, 2 4} (A difference between fold and transducers is that transducers can work in infinite input types like core.async channels) Here’s a bit of code I put together to show the relation to fold: https://github.com/fiddlerwoaroof/lisp-sandbox/blob/580980236bc55441af7d821c1f81b8c351a5474f/transduce-steps.lisp#L26 https://github.com/fiddlerwoaroof/lisp-sandbox/blob/58098023... The other way I think of this is “explicit stream fusion”: transducers work by passing an explicit continuation around and you call that continuation to add results to the output: but, crucially, you don’t have to know what the output type is, you just say “add this to it”.
• curuinor 3y ago
  It seems folks want a working example. Here's one in prod: Metabase is a BI tool, backend written mostly in Clojure. Like basically all BI tools they have this intermediate representation language thing so you write the same thing in "MBQL (metabase query language)" and it theoretically becomes same query in like, Postgres and Mongo and whatever. End user does not usually write MBQL, it's a service for the frontend querybuilding UI thing and lots of other frontend UI stuff mainly in usage. Whole processing from MBQL -> your SQL or whatever is done via a buncha big-ass transducers. There's a query cache and lots of other stuff, you need state, but you also need it to be basically fast. Metabase is not materially faster than other BI tools (because all the other BI tools do something vaguely similar in their langs and because the limiting factor is still the actual query running in most cases) but it's pretty comparable speed and the whole thing was materially written by like 5 peeps. https://github.com/metabase/metabase/blob/master/src/metabase/query_processor.clj https://github.com/metabase/metabase/blob/master/src/metabas... (nb: I used to work for Metabase but currently do not. but open core is open core)
• talkingtab 3y ago
  Sometimes sequence programming logic is easier to write and understand. Network streams being an example for me. The benefit I see of transducers is that it allows you to think with that model. For example if I want to find all image files on my computer, remove duplicates, etc. The processing is much easier for me to think about if I can think of it as one file at a time. Then some more sequential programming based on other characteristics of the file - like batching where and when it was taken. I like the concept so much that I built a JavaScript version using generators. Unfortunately, I do am not a fan of JavaScript at all and it appears that clojure now requires that I install a JVM. I realize that this is a very personal issue, but I am a "just say no to Java" person after using it for 10 years.
  • Capricorn2481 3y ago
    ?? Clojure has always run on the JVM
  • kaba0 3y ago
    Why would you say no to Java? Even if you dislike the language, it is a tool and we are supposedly professionals.
    • talkingtab 3y ago
      I realize this is controversial, and am not saying it is even rational. I spent years writing Java. And at the end I just decided that I could have written the same stuff in maybe 1/4 of the time. I understand that some people see benefits from all the tedium of it, but in the end I just felt like I had wasted and enormous amount of time. The cost was way high compared to the benefit. My other reference is just as personal and perhaps just as irrational. Java was free to use. Then after many people and companies used this free tool, Oracle came along and wanted to exploit the fact that people had an investment in this. Yes. I know. Oracle can do what they like. They own it. But I can do what I like and that is not to use anything associated with Oracle. Ever. (EDIT: and yes I do know there are open source JVM's) So, no. I guess I am not professional.
      • 3y ago
• 3cats-in-a-coat 3y ago
  I don't have experience with Clojure, but this description makes transducers sound like simple pipeline of functions transforming a stream along the way. This is quite common and readily available in any language, it's also how Unix piping works (for text/binary, but that's a stream of chars/bytes), and I'm wondering why was the name "transducer" required.
  • dataangel 3y ago
    exactly, I see nothing new here
    • the-alchemist 3y ago
      The ideas are similar, but transducers all greater flexibility and composability and testability. The pure idea of "pipelines" isn't anything new, though, as you pointed out. When you have a Unix pipeline, let's say "cat file | grep -v '^#' | sed 's/\/t//' | wc -l", or something, each step must specify where the input and output comes from. i.e., you can't "pull out" the grep, sed, and wc parts, and then tell it that it's coming from a network socket, or web service instead. It _must_ come from a file descriptor (stdin, stdout). And you can't (easily) multi-thread just the sed piece. Or (easily) add a logger between cat and grep. Or a retry between two parts. Or have it read from a network socket instead. And it's not simple to combine these pipelines with other complex pipelines. Anyway, take that concept, run with it for a while, and you get transducers.
• ducktective 3y ago
  >For the noncognoscenti... Expected nothing less from the legendary Clojure linguistics aficionados!
• colingw 3y ago
  If you're looking for Transducers in other Lisps, I'm working on a porting-and-API unification project here: https://sr.ht/~fosskers/transducers/sources https://sr.ht/~fosskers/transducers/sources Note that the Emacs Lisp variant is still under development. Once things looks good, we'd like to circle back around to Scheme's SRFI-171 (mentioned elsewhere here) and possibly redo it to include more fundamental primitives.
• miahwilde 3y ago
  # The Belt. def belt(): # return itertools.cycle([1,2,3,4,5]) x = 1 while True: yield x x += 1 if x > 5: x = 1 # The +1 def add1(iterable): for x in iterable: yield x + 1 # The +y def add(y): def _(iterable): for x in iterable: yield x + y return _ # The Oddifier def oddonly(iterable): for x in iterable: if x % 2 == 1: yield x # The Reaper def partition(n): assert n > 0 def _(iterable): batch = [] for x in iterable: batch.append(x) if len(batch) == n: yield batch batch = [] if batch: yield batch return _ # The Composer def compose(iterable, t1, t2): yield from t2(t1(iterable)) # The Plumber def pipe(iterable, p): for xform in p: iterable = xform(iterable) yield from iterable # The Press. def printall(iterator): import time for x in iterator: print(x, end=", ", flush=True) time.sleep(0.3) # printall(pipe(belt(), [add(1)])) # 2, 3, 4, 5, 6, 2, 3, 4... # printall(pipe(belt(), [add(1), oddonly])) # 3, 5, 3, 5, ... printall(pipe(belt(), [add(1), oddonly, partition(3)])) # [3, 5, 3], [5, 3, 5], ...
• devin 3y ago
  Rich gave a talk on this, and there is an associated HN thread for the talk, but some of you may be interested in the original transducers presentation. You can view it here: https://www.youtube.com/watch?v=6mTbuzafcII https://www.youtube.com/watch?v=6mTbuzafcII
• mirekrusin 3y ago
  For js/ts developer the best explanation is - you have a functions that take iterable on input and returns generator. If you arrange your functions as higher order functions (functions that return Iterable => Generator function) you can pipe them constructing arbitrary pipelines. We use this technique in trading systems. It works very well for certain class of problems. It looks like this [0]. [0] https://observablehq.com/@mirek/project-euler https://observablehq.com/@mirek/project-euler
• alexvitkov 3y ago
  I will use your language, but no offense, if I see this in something I have to maintain I'm rewriting it. (defn transformed-belt [xf] (let [ch (chan 1 xf)] (thread (loop [] (when-some [value (<!! ch)] (println "Value:" (pr-str value))) (recur))) ch)) No clue what it means, but I'm convinced it can be written in 3 lines with a for loop in a way that 100% of people looking it will understand it, and not 5%. Probably even in Clojure!
  • deleted 3y ago
    [deleted]
  • mksybr 3y ago
    I'm not expert Clojurist, but my first guess of this is that it prints and pushes every non-nil value into a go channel xf, one-by-one, recursively, in a new thread, then return the channel, presumably so more work can be done and passed to it. The number of exclamations in <!! means something, but I forget what, I think something about non/blocking.
  • adityaathalye 3y ago
    What design choice / contract do you think you will break if you were to unilaterally rewrite such a thing?