4 ms·

Not sure I understand - so Clojure is getting first class support for lazy collections and curried combinators? Or am I missing the important part?

_halgari 12y ago
Many Clojure abstractions need things like map, filter, concat, etc. Reducers are eager, lazy-seqs are lazy, and core.async channels are lazy and push-based. This is a way to unify all these abstractions so that you can write a single map/filter/concat transform once, and use it in many different ways.
- smrtinsert 12y ago
  Isn't there some overlap between this and extend-protocol? To my vague understanding this is from the functional composition angle instead of a type, but otherwise don't they accomplish the same thing?
- dustingetz 12y ago
  > This is a way to unify all these abstractions so that you can write a single map/filter/concat transform once, and use it in many different ways. I don't understand why these things aren't all unified in the first place, it's just function composition we're talking about here right? Edit: I mean it's clearly not just simple function composition, but I don't understand why not (def xform (comp (map inc) (filter even?))) ; xform is a transducer (defn xform [aseq] (->> aseq (map inc) (filter even?))) ; xform is a fn Why do these have to be different things? Why is there a need for machinery to do what fn composition is supposed to do?
  - richhickey 12y ago
    Your typical definition of map, filter etc includes concrete usage of e.g. lists. These don't. Transducers are not just currying or partial application of the map function over lists, they isolate the logic of map, filter etc from lists or any particular context, allowing them to be used in very different (e.g. non-collection) contexts.
    - kazinator 12y ago
      By "map in a non-collection context", what would be an example? Say, mapping a boolean negation function over the bits of an integer, to produce its complement? How does that look with transducers?
      - drcode 12y ago
        Well, the most obvious value to me is that you can create algorithms that defer their action- So instead of having a function that operates on a collection, you can have a function that will operate on a collection at a future date, even if the collection does not yet exist. (that's what the clojure reducer library does) (And if you say "deferring an action is just another term for 'functions'" then that's exactly the point, lifting more algorithmic logic into composable functions)
        kazinator 12y ago
        Alan Perlis Epigram #2.
    - dustingetz 12y ago
      > isolate the logic of map, filter etc from lists or any particular context so Functors (fmap)? fmap still uses regular old fns, no machinery. http://clojuredocs.org/clojure_contrib/clojure.contrib.generic.functor/fmap http://clojuredocs.org/clojure_contrib/clojure.contrib.gener... http://learnyouahaskell.com/making-our-own-types-and-typeclasses#the-functor-typeclass http://learnyouahaskell.com/making-our-own-types-and-typecla...
      - Flow 12y ago
        I feel that same confusion as you. In Haskell terms, what is it Rich have implemented or perhaps even invented?
        berdario 12y ago
        I'm not sure I grok this, but I think that the main points are: - reducers can work on tree structures, and thus can exploit parallelism. This would be like using a map that requires only the Foldable typeclass - In Haskell you have stream/vector fusion, but it's not obvious to know when ghc will actually exploit it, you might want to use something like Control.Monad.Stream or Data.Vector. In theory it might be generalized to all Foldables, but in practice for now it might be a good enough compromise to stick to a limited number of types (the one that support such transducers) So: nothing terribly new/groundbreaking, but it might bring something like stream fusion to the masses (of clojure developers :P )
- drcode 12y ago
  Yeah, when looking at core.async transformers and the reducer library, it seemed clear that there was some generality in there that could be factored out. Glad someone smarter than me figured out what it was.
jzelinskie 12y ago
I think this is just announcing that the Clojure core libraries are gaining another arity for many functions specifically for this usage so that you don't have to use (partial) or the short-hand #() syntax to use this pattern. I assume the goal is more community awareness and cleaner syntax for the re-use of this pattern.
- richhickey 12y ago
  No, (map f) is not curried map. It returns an entirely different thing - a function of reducing function to reducing function, aka a reducing function transformer, aka a transducer.
  - avernet 12y ago
    Providing the signature of this new `map` function (e.g. as in Haskell's fmal http://www.haskell.org/hoogle/?hoogle=fmap http://www.haskell.org/hoogle/?hoogle=fmap), would certainly go a long way towards helping people understand what this `map` does.
    - richhickey 12y ago
      These sigs are for the arities below only. map f: (a->b)->(x->b->x)->(x->a->x) filter pred: (a->bool)->(x->a->x)->(x->a->x) flatmap f: (a->[b])->(x->b->x)->(x->a->x) etc.
      - spion 12y ago
        So, a transduceMap implementation would be this I guess? transduceMap :: (b -> a) -> (acc -> a -> acc) -> (acc -> b -> acc) transduceMap f = \reduceFn -> \acc el -> reduceFn acc (f el) lambda added for clarity (no pun intended), however types are easier to match when using this syntax: transduceMap :: (b -> a) -> (acc -> a -> acc) -> acc -> b -> acc transduceMap f reduceFn acc el = reduceFn acc (f el)
      - avernet 12y ago
        OK, maybe we're getting somewhere. Let me try to write this `map`, just to see. I'm using Scala, so I can put some types, and have the compiler yell at me if I'm doing something overtly wrong (I need all the help I can get!). For clarity, let's define a type alias for reducers: type Reducer[X, A] = (X, A) ⇒ X Let's define `map` to match the type definition you provided. And with that type definition, I only see one way in which the function can be implemented. So it must be: def map[X, A, B](f: A ⇒ B): (Reducer[X, B] ⇒ Reducer[X, A]) = (redB: Reducer[X, B]) ⇒ (x: X, a: A) ⇒ redB(x, f(a)) How can I use this? Let's try the following: def addup(zero: Int, a: List[Int]) = a.foldLeft(zero)(_ + _) def parseList(a: List[String]) = a.map(_.toInt) map(parseList)(addup)(1, List("7", "8")) This returns 16. OK, parsing the list, and adding up starting from 1. But it doesn't look to me like `map` implements anything like the usual semantic of map. It just converts the data structure, and applies the reducer. What am I missing here?

_halgari 12y ago

Many Clojure abstractions need things like map, filter, concat, etc. Reducers are eager, lazy-seqs are lazy, and core.async channels are lazy and push-based. This is a way to unify all these abstractions so that you can write a single map/filter/concat transform once, and use it in many different ways.

smrtinsert 12y ago
Isn't there some overlap between this and extend-protocol? To my vague understanding this is from the functional composition angle instead of a type, but otherwise don't they accomplish the same thing?
dustingetz 12y ago
> This is a way to unify all these abstractions so that you can write a single map/filter/concat transform once, and use it in many different ways. I don't understand why these things aren't all unified in the first place, it's just function composition we're talking about here right? Edit: I mean it's clearly not just simple function composition, but I don't understand why not (def xform (comp (map inc) (filter even?))) ; xform is a transducer (defn xform [aseq] (->> aseq (map inc) (filter even?))) ; xform is a fn Why do these have to be different things? Why is there a need for machinery to do what fn composition is supposed to do?
- richhickey 12y ago
  Your typical definition of map, filter etc includes concrete usage of e.g. lists. These don't. Transducers are not just currying or partial application of the map function over lists, they isolate the logic of map, filter etc from lists or any particular context, allowing them to be used in very different (e.g. non-collection) contexts.
  - kazinator 12y ago
    By "map in a non-collection context", what would be an example? Say, mapping a boolean negation function over the bits of an integer, to produce its complement? How does that look with transducers?
    - drcode 12y ago
      Well, the most obvious value to me is that you can create algorithms that defer their action- So instead of having a function that operates on a collection, you can have a function that will operate on a collection at a future date, even if the collection does not yet exist. (that's what the clojure reducer library does) (And if you say "deferring an action is just another term for 'functions'" then that's exactly the point, lifting more algorithmic logic into composable functions)
      - kazinator 12y ago
        Alan Perlis Epigram #2.
  - dustingetz 12y ago
    > isolate the logic of map, filter etc from lists or any particular context so Functors (fmap)? fmap still uses regular old fns, no machinery. http://clojuredocs.org/clojure_contrib/clojure.contrib.generic.functor/fmap http://clojuredocs.org/clojure_contrib/clojure.contrib.gener... http://learnyouahaskell.com/making-our-own-types-and-typeclasses#the-functor-typeclass http://learnyouahaskell.com/making-our-own-types-and-typecla...
    - Flow 12y ago
      I feel that same confusion as you. In Haskell terms, what is it Rich have implemented or perhaps even invented?
      - berdario 12y ago
        I'm not sure I grok this, but I think that the main points are: - reducers can work on tree structures, and thus can exploit parallelism. This would be like using a map that requires only the Foldable typeclass - In Haskell you have stream/vector fusion, but it's not obvious to know when ghc will actually exploit it, you might want to use something like Control.Monad.Stream or Data.Vector. In theory it might be generalized to all Foldables, but in practice for now it might be a good enough compromise to stick to a limited number of types (the one that support such transducers) So: nothing terribly new/groundbreaking, but it might bring something like stream fusion to the masses (of clojure developers :P )
drcode 12y ago
Yeah, when looking at core.async transformers and the reducer library, it seemed clear that there was some generality in there that could be factored out. Glad someone smarter than me figured out what it was.

jzelinskie 12y ago

I think this is just announcing that the Clojure core libraries are gaining another arity for many functions specifically for this usage so that you don't have to use (partial) or the short-hand #() syntax to use this pattern. I assume the goal is more community awareness and cleaner syntax for the re-use of this pattern.

richhickey 12y ago
No, (map f) is not curried map. It returns an entirely different thing - a function of reducing function to reducing function, aka a reducing function transformer, aka a transducer.
- avernet 12y ago
  Providing the signature of this new `map` function (e.g. as in Haskell's fmal http://www.haskell.org/hoogle/?hoogle=fmap http://www.haskell.org/hoogle/?hoogle=fmap), would certainly go a long way towards helping people understand what this `map` does.
  - richhickey 12y ago
    These sigs are for the arities below only. map f: (a->b)->(x->b->x)->(x->a->x) filter pred: (a->bool)->(x->a->x)->(x->a->x) flatmap f: (a->[b])->(x->b->x)->(x->a->x) etc.
    - spion 12y ago
      So, a transduceMap implementation would be this I guess? transduceMap :: (b -> a) -> (acc -> a -> acc) -> (acc -> b -> acc) transduceMap f = \reduceFn -> \acc el -> reduceFn acc (f el) lambda added for clarity (no pun intended), however types are easier to match when using this syntax: transduceMap :: (b -> a) -> (acc -> a -> acc) -> acc -> b -> acc transduceMap f reduceFn acc el = reduceFn acc (f el)
    - avernet 12y ago
      OK, maybe we're getting somewhere. Let me try to write this `map`, just to see. I'm using Scala, so I can put some types, and have the compiler yell at me if I'm doing something overtly wrong (I need all the help I can get!). For clarity, let's define a type alias for reducers: type Reducer[X, A] = (X, A) ⇒ X Let's define `map` to match the type definition you provided. And with that type definition, I only see one way in which the function can be implemented. So it must be: def map[X, A, B](f: A ⇒ B): (Reducer[X, B] ⇒ Reducer[X, A]) = (redB: Reducer[X, B]) ⇒ (x: X, a: A) ⇒ redB(x, f(a)) How can I use this? Let's try the following: def addup(zero: Int, a: List[Int]) = a.foldLeft(zero)(_ + _) def parseList(a: List[String]) = a.map(_.toInt) map(parseList)(addup)(1, List("7", "8")) This returns 16. OK, parsing the list, and adding up starting from 1. But it doesn't look to me like `map` implements anything like the usual semantic of map. It just converts the data structure, and applies the reducer. What am I missing here?