6 ms·

Rich Hickey – Inside Transducers [video]

112 points by tsm 12y ago

nickik 12y ago
Again a more technical talk. I really liked it. Specially the end bit about pipelines, that seams like it would be really usful. Transducers overall are quite intressting, I am exited to see what other transducer context people come up with.
- moomin 12y ago
  It's not that clear there will be one. Transducers have three steps: begin, during and end. In practice, start is only used for reduce operations and outside of reduce operations end can only be used for its side effects. There's also definitional issues e.g. can you have an async transducer?
  - cfeduke 12y ago
    Yes you can have async and blocking transducers. Forward to the point in the video about channels for a discussion.
    - moomin 12y ago
      But if you use an async transducer on a channel, it fails. (Actual tested code, sadly.) Which might be okay, but my basic point that it's definitionally fuzzy still holds.
  - kbeaty 12y ago
    You can indeed use transducers in asynchronous contexts. In fact, that is what intrigues me most about the abstraction. Transducers are defined such that you can abstract the context of input, output and iteration and focus on the transformation of each element from an independent source individually. The implementation of each transducer accepts another transformation in a "pipeline" (eventually the output sink) and accepts an input during an external iteration process. The implementation decides what to do with it (map changes with a function, filter ignores certain values with a predicate, etc.) You can also define transducers that send multiple values for each input (think string.split) or some that do not send any until completion (think buffered results). Since the iteration source and sink are abstracted from the transformation, you can use the same transformation in other contexts (Promises, event streams, IO streams, CSP, etc.) I've been experimenting with transducers in asynchronous contexts in JavaScript if anyone is interested, for example: [1]: http://simplectic.com/projects/underscore-transducer/ http://simplectic.com/projects/underscore-transducer/ [2]: http://simplectic.com/projects/underarm/ http://simplectic.com/projects/underarm/ [3]: https://github.com/transduce/transduce-async https://github.com/transduce/transduce-async [4]: https://github.com/kevinbeaty/transduce-stream https://github.com/kevinbeaty/transduce-stream [5]: https://github.com/transduce/transduce-push https://github.com/transduce/transduce-push [6]: https://github.com/transduce/transduce-then https://github.com/transduce/transduce-then
ultimape 12y ago
For those not familiar with Clojure, here's a great demonstration of the concept done up in JavaScript: http://jlongster.com/Transducers.js--A-JavaScript-Library-for-Transformation-of-Data http://jlongster.com/Transducers.js--A-JavaScript-Library-fo...
- jonahx 12y ago
  From that link: "The reduce function is the base transformation; any other transformation can be expressed in terms of it (map, filter, etc)." This seems so obvious in retrospect -- I can't believe I had never made that connection before.
  - ultimape 12y ago
    I had a similar epiphany while learning about regular expressions in Perl. That connection with text processing is what helped me understand list comprehensions. They seem strikingly similar. For JS, I personally prefer lo-dash for this kind of work, or dropping in polyfills from MDN. For some context, here are .map functions applied to normal arrays http://jsperf.com/native-vs-array-js-vs-underscore/54 http://jsperf.com/native-vs-array-js-vs-underscore/54 I've been looking for similar libraries that work on typed-arrays because they are so much more efficient when working with web-workers or with raw canvas data. My attempts at hacking it in feel like they are just bad ideas: http://jsperf.com/float32array-map/2 http://jsperf.com/float32array-map/2
    - grayrest 12y ago
      Most people use underscore/lodash for this stuff. The difference is that the js transducers libraries don't create intermediate arrays, only do enough work to produce the requested output, and work on top of anything that can be coerced into the iteration protocol. I've seen demos of using Facebook's Immutable JS, CSP.js [1], and I don't see why you couldn't put them on top of Typed Arrays or a FRP library like Kefir. [1] http://jlongster.com/s/nationjs-slides/ http://jlongster.com/s/nationjs-slides/
      - 12y ago
ultimape 12y ago
Here's another video by Rick Hickey about transducers: https://www.youtube.com/watch?v=6mTbuzafcII https://www.youtube.com/watch?v=6mTbuzafcII - it was referenced in this great post detailing the transducer notion with Clojure, Scala, and Haskell: http://blog.podsnap.com/ducers2.html http://blog.podsnap.com/ducers2.html
charlysisto 12y ago
So Transducers generalize the usage of Enumerable functions such as map, reduce, filter, flatMap (...). Please could someone tell me if this is conceptually different from ruby's Enumerable module which only needs the class its included in to implement `each` so anything can be enumerable ? Or is it a similar but just in translated to the FP world ?
- moomin 12y ago
  What's you're thinking of is more like ISeq in Clojure, or Foldable is Haskell. The more interesting generalisation is not between arrays, vectors and hashsets, but between data structures and event streams.
- YuriNiyazov 12y ago
  Yes; There are differences. 1. Transducers are parallel under the covers. Since the expectations is that the code that the predicate or mapping functions that you pass into map, filter and reduce are pure (no variables are changed, no state is modified, just a calculation that is only dependent on arguments) the parallelism is hidden away from you, but it's there. Ruby's Enumerable can't do parallelism 2. When you compose transducers, there are no intermediate sequences generated. The simplest example is this: In Ruby: [1, 2, 3].map {|x| x + 1}.map {|x| x+ 2} will generate an intermediate array [2, 3, 4] after the first map, and then will generate [4, 5, 6] when it has evaluated the whole expression. In Clojure (I am sure I got the syntax wrong for this one) (transduce (map #(+ 2 %) (map inc)) [1 2 3]) will create an intermediate mapping function that will first increment, then add 2 to its argument, and then will map once using that intermediate function.
  - cfeduke 12y ago
    Transducers _can be_ parallelized (TODO) but some transducer implementations do contain state, like `take` and therefore cannot be parallelized. The big thing is no intermediate results.
    - seanmcdirmid 12y ago
kushti 12y ago
For those who know Russian, here's awesome critics of "transreducers" http://thedeemon.livejournal.com/87320.html http://thedeemon.livejournal.com/87320.html
- vanderZwan 12y ago
  Could you try to give a summary for those who don't?
  - juskrey 12y ago
    The summary: "Hickey did not invent FP, how dares he?"
    - nickik 12y ago
      People keep attacking him whenever he interduces something. Why? Every time he releases something, he also shows from what research it originated. Go back to the first transducer talk and you will see the references.
      - lomnakkus 12y ago
        Personally, I don't mind him too much even though I'm a typeful-programming weenie. That said, he does have a tendency to somewhat unfoundedly say "... and you couldn't do this in a typed language" (or at least allude to it)... whereas people again and again show that, yes, it could be done in a statically typed language. In fact, this is trivially true in a sense as shown by Bob Harper[1] another person who is undoubtedly hugely influential, but who I have trouble with because of his obvious bias and trollish ways. (I think Rich is absolutely spot on on many of the more abstract things about or industry/craft, but this is just one of those details that irks me.) [1] http://existentialtype.wordpress.com/2011/03/19/dynamic-languages-are-static-languages/ http://existentialtype.wordpress.com/2011/03/19/dynamic-lang...
        ajanuary 12y ago
Blackthorn 12y ago
Even if you forget entirely about Clojure for a second, Rich has this very rare gift to take a complicated subject and make it easy for the audience to understand.
- lomnakkus 12y ago
  This is a double-edged sword. Forgive me a small anecdote: I once had a wonderful tutor who could explain any advanced concept in highly intuitive terms and it just made sense... until I got out of the classroom. At which point I'd just have this feeling of "Hang on, whaaaaaa...?". The first time I attempted the exam in this particular subject matter, I failed miserably (rightly so). Having learned my lesson, I went back to study the actual source material more thoroughly instead of mostly just listening to the probably-best-educator on the subject. That time I actually understood the material and passed with flying colours. (I still think the particular educator had a major role in me passing at all, but I digress.) That that for what you will, but please be aware that a "gift for simplification" sometimes is a double-edged sword and can leave the audience thinking that they've understood when, actually, they haven't. I'm not saying that's the case here, just something to be wary of.
  - agumonkey 12y ago
    I had similar experiences many times, and I don't know what's best, looking at an abstraction for weeks until you get the "ohh they just meant this" or knowing in advance it's not that obscure and then work to imprint said abstraction deeper.
    - lomnakkus 12y ago
      Ever since that experience, I've tried to somehow try to practice working with "X" in some concrete way, even though it might feel like a classic "math problem" situation. Personally, I find it helps me get a grasp on "X", whatever it might be. (Mind you, this is just personal experience so YMMV.)
  - brudgers 12y ago
    The difference here is that Rich Hickey is explaining a fundamental computing mechanism that many programmers have come in with but few think about enough to give a name: the finite state transducer [see Wikipedia]. The name "transducers" is not an invention, it's basic computer science. The idea of using transducers to abstract over an input is not really controversial in the sense that it's the essence of what turns source into executable code on digital computers. Sure, Ther are tradeoffs relative to which flavor of Turing machine underpins any abstraction, but at least this is an area where we can achieve clarity. This is something Hickey implemented, not invented. Like all of automata, understanding the concept is likely non-trivial for some pretty smart people since it requires thinking about first principles of computing.
sitkack 12y ago
These aren't materially different from itertools in Python.

112 points by tsm 12y ago

Again a more technical talk. I really liked it. Specially the end bit about pipelines, that seams like it would be really usful. Transducers overall are quite intressting, I am exited to see what other transducer context people come up with.

moomin 12y ago
It's not that clear there will be one. Transducers have three steps: begin, during and end. In practice, start is only used for reduce operations and outside of reduce operations end can only be used for its side effects. There's also definitional issues e.g. can you have an async transducer?
- cfeduke 12y ago
  Yes you can have async and blocking transducers. Forward to the point in the video about channels for a discussion.
  - moomin 12y ago
    But if you use an async transducer on a channel, it fails. (Actual tested code, sadly.) Which might be okay, but my basic point that it's definitionally fuzzy still holds.
- kbeaty 12y ago
  You can indeed use transducers in asynchronous contexts. In fact, that is what intrigues me most about the abstraction. Transducers are defined such that you can abstract the context of input, output and iteration and focus on the transformation of each element from an independent source individually. The implementation of each transducer accepts another transformation in a "pipeline" (eventually the output sink) and accepts an input during an external iteration process. The implementation decides what to do with it (map changes with a function, filter ignores certain values with a predicate, etc.) You can also define transducers that send multiple values for each input (think string.split) or some that do not send any until completion (think buffered results). Since the iteration source and sink are abstracted from the transformation, you can use the same transformation in other contexts (Promises, event streams, IO streams, CSP, etc.) I've been experimenting with transducers in asynchronous contexts in JavaScript if anyone is interested, for example: [1]: http://simplectic.com/projects/underscore-transducer/ http://simplectic.com/projects/underscore-transducer/ [2]: http://simplectic.com/projects/underarm/ http://simplectic.com/projects/underarm/ [3]: https://github.com/transduce/transduce-async https://github.com/transduce/transduce-async [4]: https://github.com/kevinbeaty/transduce-stream https://github.com/kevinbeaty/transduce-stream [5]: https://github.com/transduce/transduce-push https://github.com/transduce/transduce-push [6]: https://github.com/transduce/transduce-then https://github.com/transduce/transduce-then

ultimape 12y ago

For those not familiar with Clojure, here's a great demonstration of the concept done up in JavaScript: http://jlongster.com/Transducers.js--A-JavaScript-Library-for-Transformation-of-Data http://jlongster.com/Transducers.js--A-JavaScript-Library-fo...

jonahx 12y ago
From that link: "The reduce function is the base transformation; any other transformation can be expressed in terms of it (map, filter, etc)." This seems so obvious in retrospect -- I can't believe I had never made that connection before.
- ultimape 12y ago
  I had a similar epiphany while learning about regular expressions in Perl. That connection with text processing is what helped me understand list comprehensions. They seem strikingly similar. For JS, I personally prefer lo-dash for this kind of work, or dropping in polyfills from MDN. For some context, here are .map functions applied to normal arrays http://jsperf.com/native-vs-array-js-vs-underscore/54 http://jsperf.com/native-vs-array-js-vs-underscore/54 I've been looking for similar libraries that work on typed-arrays because they are so much more efficient when working with web-workers or with raw canvas data. My attempts at hacking it in feel like they are just bad ideas: http://jsperf.com/float32array-map/2 http://jsperf.com/float32array-map/2
  - grayrest 12y ago
    Most people use underscore/lodash for this stuff. The difference is that the js transducers libraries don't create intermediate arrays, only do enough work to produce the requested output, and work on top of anything that can be coerced into the iteration protocol. I've seen demos of using Facebook's Immutable JS, CSP.js [1], and I don't see why you couldn't put them on top of Typed Arrays or a FRP library like Kefir. [1] http://jlongster.com/s/nationjs-slides/ http://jlongster.com/s/nationjs-slides/
    - 12y ago

ultimape 12y ago

Here's another video by Rick Hickey about transducers: https://www.youtube.com/watch?v=6mTbuzafcII https://www.youtube.com/watch?v=6mTbuzafcII - it was referenced in this great post detailing the transducer notion with Clojure, Scala, and Haskell: http://blog.podsnap.com/ducers2.html http://blog.podsnap.com/ducers2.html

charlysisto 12y ago

So Transducers generalize the usage of Enumerable functions such as map, reduce, filter, flatMap (...). Please could someone tell me if this is conceptually different from ruby's Enumerable module which only needs the class its included in to implement `each` so anything can be enumerable ? Or is it a similar but just in translated to the FP world ?

moomin 12y ago
What's you're thinking of is more like ISeq in Clojure, or Foldable is Haskell. The more interesting generalisation is not between arrays, vectors and hashsets, but between data structures and event streams.
YuriNiyazov 12y ago
Yes; There are differences. 1. Transducers are parallel under the covers. Since the expectations is that the code that the predicate or mapping functions that you pass into map, filter and reduce are pure (no variables are changed, no state is modified, just a calculation that is only dependent on arguments) the parallelism is hidden away from you, but it's there. Ruby's Enumerable can't do parallelism 2. When you compose transducers, there are no intermediate sequences generated. The simplest example is this: In Ruby: [1, 2, 3].map {|x| x + 1}.map {|x| x+ 2} will generate an intermediate array [2, 3, 4] after the first map, and then will generate [4, 5, 6] when it has evaluated the whole expression. In Clojure (I am sure I got the syntax wrong for this one) (transduce (map #(+ 2 %) (map inc)) [1 2 3]) will create an intermediate mapping function that will first increment, then add 2 to its argument, and then will map once using that intermediate function.
- cfeduke 12y ago
  Transducers _can be_ parallelized (TODO) but some transducer implementations do contain state, like `take` and therefore cannot be parallelized. The big thing is no intermediate results.
  - seanmcdirmid 12y ago

kushti 12y ago

For those who know Russian, here's awesome critics of "transreducers" http://thedeemon.livejournal.com/87320.html http://thedeemon.livejournal.com/87320.html

vanderZwan 12y ago
Could you try to give a summary for those who don't?
- juskrey 12y ago
  The summary: "Hickey did not invent FP, how dares he?"
  - nickik 12y ago
    People keep attacking him whenever he interduces something. Why? Every time he releases something, he also shows from what research it originated. Go back to the first transducer talk and you will see the references.
    - lomnakkus 12y ago
      Personally, I don't mind him too much even though I'm a typeful-programming weenie. That said, he does have a tendency to somewhat unfoundedly say "... and you couldn't do this in a typed language" (or at least allude to it)... whereas people again and again show that, yes, it could be done in a statically typed language. In fact, this is trivially true in a sense as shown by Bob Harper[1] another person who is undoubtedly hugely influential, but who I have trouble with because of his obvious bias and trollish ways. (I think Rich is absolutely spot on on many of the more abstract things about or industry/craft, but this is just one of those details that irks me.) [1] http://existentialtype.wordpress.com/2011/03/19/dynamic-languages-are-static-languages/ http://existentialtype.wordpress.com/2011/03/19/dynamic-lang...
      - ajanuary 12y ago

Blackthorn 12y ago

Even if you forget entirely about Clojure for a second, Rich has this very rare gift to take a complicated subject and make it easy for the audience to understand.

lomnakkus 12y ago
This is a double-edged sword. Forgive me a small anecdote: I once had a wonderful tutor who could explain any advanced concept in highly intuitive terms and it just made sense... until I got out of the classroom. At which point I'd just have this feeling of "Hang on, whaaaaaa...?". The first time I attempted the exam in this particular subject matter, I failed miserably (rightly so). Having learned my lesson, I went back to study the actual source material more thoroughly instead of mostly just listening to the probably-best-educator on the subject. That time I actually understood the material and passed with flying colours. (I still think the particular educator had a major role in me passing at all, but I digress.) That that for what you will, but please be aware that a "gift for simplification" sometimes is a double-edged sword and can leave the audience thinking that they've understood when, actually, they haven't. I'm not saying that's the case here, just something to be wary of.
- agumonkey 12y ago
  I had similar experiences many times, and I don't know what's best, looking at an abstraction for weeks until you get the "ohh they just meant this" or knowing in advance it's not that obscure and then work to imprint said abstraction deeper.
  - lomnakkus 12y ago
    Ever since that experience, I've tried to somehow try to practice working with "X" in some concrete way, even though it might feel like a classic "math problem" situation. Personally, I find it helps me get a grasp on "X", whatever it might be. (Mind you, this is just personal experience so YMMV.)
- brudgers 12y ago
  The difference here is that Rich Hickey is explaining a fundamental computing mechanism that many programmers have come in with but few think about enough to give a name: the finite state transducer [see Wikipedia]. The name "transducers" is not an invention, it's basic computer science. The idea of using transducers to abstract over an input is not really controversial in the sense that it's the essence of what turns source into executable code on digital computers. Sure, Ther are tradeoffs relative to which flavor of Turing machine underpins any abstraction, but at least this is an area where we can achieve clarity. This is something Hickey implemented, not invented. Like all of automata, understanding the concept is likely non-trivial for some pretty smart people since it requires thinking about first principles of computing.

sitkack 12y ago

These aren't materially different from itertools in Python.