6 ms·

To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of tr

by casion 3y ago

To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. So rather than processing the collection, passing it to the next function that processes the collection, passing it to the next... etc.. consuming all the CPU and memory that involves, you can define steps that are applied for each item in the collection thereby having the iteration through the collection happen once. These steps (transducers) are also composable and reusable. I suspect you know this, consider this a basic explanation for other people reading.

tyre 3y ago
> To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. This is a much better and clearer explanation than the entire article.
mrkeen 3y ago
That seems like insufficient magic for the respect that transducers seem to have. What you've written is just the Haskell list monad or Java8 Stream.flatmap.
- waffletower 3y ago
  I think this opinion is less nuanced than it could be. Perhaps this will help? https://hypirion.com/musings/haskell-transducers https://hypirion.com/musings/haskell-transducers
  - lgas 3y ago
    Surely the example that says these two would be equivalent is wrong: foldl (+) 0 (take n myList) reduce (takeNPlus 10) 0 myList I assume it should be (takeNPlus n)? (or (take 10 myList))
- BoiledCabbage 3y ago
  > That seems like insufficient magic for the respect that transducers seem to have. This is 100% correct. It's amazing how over hyped they are. If you have ever used C#'s LINQ you are using transducers. The fact that LINQ works item at a time instead of collection at a time is all that's being discussed. That if you say take the first two in a 1,000,000 long collection LINQ will only enumerate the first 2 items and not all 1,000,000 is the other behavior. And the way to do this is by composing operations using a "." operator into a "query" to run. C# had had this since 2007. It doesn't require a fancy name, it doesn't require streams, it doesn't require "thought pieces" every few months for over a decade for people to "master thinking in linq". Just sequence your operations and get back to coding. And Clojure is a really great language, but the amount of mental space occupied by transducers is a bad look for the language. Via analogy it's like watching people be amazed by for loops for over a decade. And I know they're are a lot of really smart people in the clojure community, so I honestly put it on Rich. Either on hyping or up so much when he released it like he just invented sliced bread and it's a deep advanced topic, or for how it's presented in the language that people have to understand so much beneath the abstraction layer to use it correctly. Your average blub enterprise programmer has been using LINQ for 15 years and never needed 100 thought pieces in how to use it and reason about it. Yes it's a monad, yes it let's your short circuit, yes it's item at a time, but a user doesn't need to know lots of detail to use it. It's like watching a language community that can do calculus be continuously hypong on the fact it can do long division. Clojure is an amazing language, transducers are not that special, figure out why they are so hyped in clojure. Or maybe I have it backwards and linq/transducers really are partial differential equations and C# snuck it into the 4th grade curriculum and nobody noticed.
  - User23 3y ago
    > If you have ever used C#'s LINQ you are using transducers. The fact that LINQ works item at a time instead of collection at a time is all that's being discussed. That if you say take the first two in a 1,000,000 long collection LINQ will only enumerate the first 2 items and not all 1,000,000 is the other behavior. And the way to do this is by composing operations using a "." operator into a "query" to run. Isn't that just function composition to build the map function? I guess that where the magic can come in is that function composition is associative, which allows for some really interesting opportunities for runtime optimization that, so far as I know, haven't been seriously explored.
- casion 3y ago
  Yes, and like most language features it's not about the feature, it's about having _that feature_ in a language with other benefits. Think generics in Go or concurrency (effects) in OCAML or smart pointers in Rust. Not at all unique things, but having them in the language with other benefits is worth some discussion as it may provide extra leverage in context.
vaylian 3y ago
If I understand you correctly, transduces transform something like (map fn1 (map fn2 (map fn3 collection))) into (map (fn [x] (fn1 (fn2 (fn3 x)))) collection); Is that correct? More idiomatic clojure: (map (comp fn1 fn2 fn3) collection)
- casion 3y ago
  You have the idea, yes.
- thom 3y ago
  It’s the mapping itself that can be composed with transducers. Where before you had pipelines of map/filter/whatever, now you have a single function representing the sequence operations, which can be used for any kind of sequence (a list in memory, or items coming in over a channel or message queue) item by item.
- sundarurfriend 3y ago
  That sort of reminds of broadcast fusion [1] in Julia. Funnily enough, Julia is where I came across the term transducers too, via the Transducers.jl package [2]. The article and the comments here now make me wonder what the difference is, between broadcast fusion and transducers. [1] https://julialang.org/blog/2017/01/moredots/ https://julialang.org/blog/2017/01/moredots/ [2] https://github.com/JuliaFolds2/Transducers.jl/ https://github.com/JuliaFolds2/Transducers.jl/
- roguas 3y ago
  Generally, yes, but I also think the fn1,fn2,fn3 can happen independently (which is why its powerful). So items of collection may be at different steps.

by casion 3y ago

tyre 3y ago

> To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. This is a much better and clearer explanation than the entire article.

mrkeen 3y ago

That seems like insufficient magic for the respect that transducers seem to have. What you've written is just the Haskell list monad or Java8 Stream.flatmap.

waffletower 3y ago
I think this opinion is less nuanced than it could be. Perhaps this will help? https://hypirion.com/musings/haskell-transducers https://hypirion.com/musings/haskell-transducers
- lgas 3y ago
  Surely the example that says these two would be equivalent is wrong: foldl (+) 0 (take n myList) reduce (takeNPlus 10) 0 myList I assume it should be (takeNPlus n)? (or (take 10 myList))
BoiledCabbage 3y ago
> That seems like insufficient magic for the respect that transducers seem to have. This is 100% correct. It's amazing how over hyped they are. If you have ever used C#'s LINQ you are using transducers. The fact that LINQ works item at a time instead of collection at a time is all that's being discussed. That if you say take the first two in a 1,000,000 long collection LINQ will only enumerate the first 2 items and not all 1,000,000 is the other behavior. And the way to do this is by composing operations using a "." operator into a "query" to run. C# had had this since 2007. It doesn't require a fancy name, it doesn't require streams, it doesn't require "thought pieces" every few months for over a decade for people to "master thinking in linq". Just sequence your operations and get back to coding. And Clojure is a really great language, but the amount of mental space occupied by transducers is a bad look for the language. Via analogy it's like watching people be amazed by for loops for over a decade. And I know they're are a lot of really smart people in the clojure community, so I honestly put it on Rich. Either on hyping or up so much when he released it like he just invented sliced bread and it's a deep advanced topic, or for how it's presented in the language that people have to understand so much beneath the abstraction layer to use it correctly. Your average blub enterprise programmer has been using LINQ for 15 years and never needed 100 thought pieces in how to use it and reason about it. Yes it's a monad, yes it let's your short circuit, yes it's item at a time, but a user doesn't need to know lots of detail to use it. It's like watching a language community that can do calculus be continuously hypong on the fact it can do long division. Clojure is an amazing language, transducers are not that special, figure out why they are so hyped in clojure. Or maybe I have it backwards and linq/transducers really are partial differential equations and C# snuck it into the 4th grade curriculum and nobody noticed.
- User23 3y ago
  > If you have ever used C#'s LINQ you are using transducers. The fact that LINQ works item at a time instead of collection at a time is all that's being discussed. That if you say take the first two in a 1,000,000 long collection LINQ will only enumerate the first 2 items and not all 1,000,000 is the other behavior. And the way to do this is by composing operations using a "." operator into a "query" to run. Isn't that just function composition to build the map function? I guess that where the magic can come in is that function composition is associative, which allows for some really interesting opportunities for runtime optimization that, so far as I know, haven't been seriously explored.
casion 3y ago
Yes, and like most language features it's not about the feature, it's about having _that feature_ in a language with other benefits. Think generics in Go or concurrency (effects) in OCAML or smart pointers in Rust. Not at all unique things, but having them in the language with other benefits is worth some discussion as it may provide extra leverage in context.

vaylian 3y ago

If I understand you correctly, transduces transform something like (map fn1 (map fn2 (map fn3 collection))) into (map (fn [x] (fn1 (fn2 (fn3 x)))) collection); Is that correct? More idiomatic clojure: (map (comp fn1 fn2 fn3) collection)

casion 3y ago
You have the idea, yes.
thom 3y ago
It’s the mapping itself that can be composed with transducers. Where before you had pipelines of map/filter/whatever, now you have a single function representing the sequence operations, which can be used for any kind of sequence (a list in memory, or items coming in over a channel or message queue) item by item.
sundarurfriend 3y ago
That sort of reminds of broadcast fusion [1] in Julia. Funnily enough, Julia is where I came across the term transducers too, via the Transducers.jl package [2]. The article and the comments here now make me wonder what the difference is, between broadcast fusion and transducers. [1] https://julialang.org/blog/2017/01/moredots/ https://julialang.org/blog/2017/01/moredots/ [2] https://github.com/JuliaFolds2/Transducers.jl/ https://github.com/JuliaFolds2/Transducers.jl/
roguas 3y ago
Generally, yes, but I also think the fn1,fn2,fn3 can happen independently (which is why its powerful). So items of collection may be at different steps.