10 ms·

What is generally missing from this category of article is a motivating statement, eg here is a problem that is easier or at least different (transformed into a

by mcbrit 3y ago

What is generally missing from this category of article is a motivating statement, eg here is a problem that is easier or at least different (transformed into a different category of problem) given this idea. Up top. When I don’t see this up top or scanning forward I assume the article assumes knowledge that I do not have, and I bounce.

adityaathalye 3y ago
True, though I think this is part personal note, and part intended as additional material for people already trying to apply transducers effectively. The author suggests this by casting it as "my" mental model, in a work context. Maybe this variant of explanation will suit your context better: https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojure/index.html#transducery-buzz https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojur... I wrote the post as a way to explore Clojure's standard library using FizzBuzz as a device.
- mcbrit 3y ago
  Most folks on HN who are interested in PLs, including me, are familiar with transducers at a high level. Composable, performant, yada yada ya. What we (or maybe just I) do not have is a nontrivial example of advantage. Your comment sharpened that for me. I don't want FizzBuzz; I want someone taking a reasonable toy problem, such as a trad+photon+quadtree raytracer and demonstrating advantage by applying the concept.
casion 3y ago
To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. So rather than processing the collection, passing it to the next function that processes the collection, passing it to the next... etc.. consuming all the CPU and memory that involves, you can define steps that are applied for each item in the collection thereby having the iteration through the collection happen once. These steps (transducers) are also composable and reusable. I suspect you know this, consider this a basic explanation for other people reading.
- tyre 3y ago
  > To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. This is a much better and clearer explanation than the entire article.
- mrkeen 3y ago
  That seems like insufficient magic for the respect that transducers seem to have. What you've written is just the Haskell list monad or Java8 Stream.flatmap.
  - waffletower 3y ago
    I think this opinion is less nuanced than it could be. Perhaps this will help? https://hypirion.com/musings/haskell-transducers https://hypirion.com/musings/haskell-transducers
    - lgas 3y ago
      Surely the example that says these two would be equivalent is wrong: foldl (+) 0 (take n myList) reduce (takeNPlus 10) 0 myList I assume it should be (takeNPlus n)? (or (take 10 myList))
  - BoiledCabbage 3y ago
    > That seems like insufficient magic for the respect that transducers seem to have. This is 100% correct. It's amazing how over hyped they are. If you have ever used C#'s LINQ you are using transducers. The fact that LINQ works item at a time instead of collection at a time is all that's being discussed. That if you say take the first two in a 1,000,000 long collection LINQ will only enumerate the first 2 items and not all 1,000,000 is the other behavior. And the way to do this is by composing operations using a "." operator into a "query" to run. C# had had this since 2007. It doesn't require a fancy name, it doesn't require streams, it doesn't require "thought pieces" every few months for over a decade for people to "master thinking in linq". Just sequence your operations and get back to coding. And Clojure is a really great language, but the amount of mental space occupied by transducers is a bad look for the language. Via analogy it's like watching people be amazed by for loops for over a decade. And I know they're are a lot of really smart people in the clojure community, so I honestly put it on Rich. Either on hyping or up so much when he released it like he just invented sliced bread and it's a deep advanced topic, or for how it's presented in the language that people have to understand so much beneath the abstraction layer to use it correctly. Your average blub enterprise programmer has been using LINQ for 15 years and never needed 100 thought pieces in how to use it and reason about it. Yes it's a monad, yes it let's your short circuit, yes it's item at a time, but a user doesn't need to know lots of detail to use it. It's like watching a language community that can do calculus be continuously hypong on the fact it can do long division. Clojure is an amazing language, transducers are not that special, figure out why they are so hyped in clojure. Or maybe I have it backwards and linq/transducers really are partial differential equations and C# snuck it into the 4th grade curriculum and nobody noticed.
javajosh 3y ago
I put together a transducer in JavaScript with the intent to simplify Redux/flux architected front-ends. Rather than have synthetic actions that trigger reduction, I allow the caller to simply add objects directly into the store, and rely on them being combined in a useful way. And rather than leave it purely at dead data, I allow you to add functions, too, such that once added they can handle subsequent input. My `combine()` is a recursive `Object.assign()` plus reification of function calls: https://simpatico.io/combine2 https://simpatico.io/combine2. It's that second part that makes it a transducer. This work predates the term "transducer" and I was happy to see Rich Hickey defining it, so I use the term retroactively. Note that I don't see much use for transducers as a general programming technique. This one use is very specific, but I have never before or since needed one.
- vanderZwan 3y ago
  > This work predates the term "transducer" and I was happy to see Rich Hickey defining it IIRC Hickey did not define this term (nor claimed to have done so), he found it in existing compsci research papers that approached them from a mathematical proof angle and realized they solved a problem he needed to solve (basically not allocating intermediate collections and doing wasteful work on parts of the data that will be immediately discarded). I think one of the authors of the papers he cited even gave a talk at a Clojure conference once.
- eyelidlessness 3y ago
  FYI: I was curious and clicked your link, but it fully reloads the page every 2-3 seconds so I had to bail (iOS Safari if that helps)
  - javajosh 3y ago
    Thanks for the feedback, fixed. I like to use a simple method to speed up my BTD loop when doing webapp work, which is to add a `meta refresh` tag. But that's inappropriate for deployment. Note that my site is very strict about 3rd party content (none is allowed), and about privacy (the only log is in a tmux buffer), so there's no reason for this behavior other than my forgetfulness.
sesm 3y ago
Initial problem was duplication of sequence library for CSP channels (core.async in Clojure). When the idea of ‘transducers’ (transformers of reducing function) was discovered, it turned out to have many additional useful properties and was integrated deeply into standard library.
- thom 3y ago
  I’ve always felt any mention of reducers is a big distraction. They’re not really key to the idea of transducers which are almost entirely about pure sequence to sequence transforms.
  - thom 3y ago
    For the benefit of downvoters, let me explain myself: the 'reducing function' (also called a 'reducing step function' in other parts of the docs) isn't always what you'd think of as the normal arity-2 function passed to reduce itself. Take a look at the reducing functions in clojure.core - the 3-arity, innermost function returned inside all the transducers. They almost never do anything with result which would be the accumulator in a normal reduction - they just pass it on or return it. In fact, it wouldn't make much sense for transducers to do any sort of reduction because if they did, you'd no longer be doing a step-wise transform over a sequence, and you could just use normal function composition because you'd have a single value to pass forward. The reduced mechanism is only really used in transducers to short circuit, so even there the name clouds the meaning. The only time that something we'd recognise as reduction is happening is when you call transduce and specify a reducing function as its f argument. You'll have seen lots of examples with + as the reducing function because it handily returns 0 when called with no args, acts as identity with 1 arg, and reduces (actually reduces!) with two args. But instead look at sequence which can take an xform. It's implemented in Java code, but the inner reducing function it uses completely ignores the accumulator value! Everywhere you're using result inside a reducing function? It's just null and ignored. But sequence elsewhere stashes away the (transformed) output of the transducer stack and lets you have it back as a sequence. No reducing is happening, just transforming sequences. Elsewhere, into supports an xform argument - what's its reducing function? It's just conjing stuff back together! So to me, the interesting bits of transducers are that they give you an efficient and composable way of creating sequence-to-sequence transformations. Any and all mentions of reducers are necessary plumbing but slightly distracting from the core ideas.

by mcbrit 3y ago

adityaathalye 3y ago

True, though I think this is part personal note, and part intended as additional material for people already trying to apply transducers effectively. The author suggests this by casting it as "my" mental model, in a work context. Maybe this variant of explanation will suit your context better: https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojure/index.html#transducery-buzz https://www.evalapply.org/posts/n-ways-to-fizzbuzz-in-clojur... I wrote the post as a way to explore Clojure's standard library using FizzBuzz as a device.

mcbrit 3y ago
Most folks on HN who are interested in PLs, including me, are familiar with transducers at a high level. Composable, performant, yada yada ya. What we (or maybe just I) do not have is a nontrivial example of advantage. Your comment sharpened that for me. I don't want FizzBuzz; I want someone taking a reasonable toy problem, such as a trad+photon+quadtree raytracer and demonstrating advantage by applying the concept.

casion 3y ago

To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. So rather than processing the collection, passing it to the next function that processes the collection, passing it to the next... etc.. consuming all the CPU and memory that involves, you can define steps that are applied for each item in the collection thereby having the iteration through the collection happen once. These steps (transducers) are also composable and reusable. I suspect you know this, consider this a basic explanation for other people reading.

tyre 3y ago
> To give a shallow overview, transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections. This is a much better and clearer explanation than the entire article.
mrkeen 3y ago
That seems like insufficient magic for the respect that transducers seem to have. What you've written is just the Haskell list monad or Java8 Stream.flatmap.
- waffletower 3y ago
  I think this opinion is less nuanced than it could be. Perhaps this will help? https://hypirion.com/musings/haskell-transducers https://hypirion.com/musings/haskell-transducers
  - lgas 3y ago
    Surely the example that says these two would be equivalent is wrong: foldl (+) 0 (take n myList) reduce (takeNPlus 10) 0 myList I assume it should be (takeNPlus n)? (or (take 10 myList))
- BoiledCabbage 3y ago
  > That seems like insufficient magic for the respect that transducers seem to have. This is 100% correct. It's amazing how over hyped they are. If you have ever used C#'s LINQ you are using transducers. The fact that LINQ works item at a time instead of collection at a time is all that's being discussed. That if you say take the first two in a 1,000,000 long collection LINQ will only enumerate the first 2 items and not all 1,000,000 is the other behavior. And the way to do this is by composing operations using a "." operator into a "query" to run. C# had had this since 2007. It doesn't require a fancy name, it doesn't require streams, it doesn't require "thought pieces" every few months for over a decade for people to "master thinking in linq". Just sequence your operations and get back to coding. And Clojure is a really great language, but the amount of mental space occupied by transducers is a bad look for the language. Via analogy it's like watching people be amazed by for loops for over a decade. And I know they're are a lot of really smart people in the clojure community, so I honestly put it on Rich. Either on hyping or up so much when he released it like he just invented sliced bread and it's a deep advanced topic, or for how it's presented in the language that people have to understand so much beneath the abstraction layer to use it correctly. Your average blub enterprise programmer has been using LINQ for 15 years and never needed 100 thought pieces in how to use it and reason about it. Yes it's a monad, yes it let's your short circuit, yes it's item at a time, but a user doesn't need to know lots of detail to use it. It's like watching a language community that can do calculus be continuously hypong on the fact it can do long division. Clojure is an amazing language, transducers are not that special, figure out why they are so hyped in clojure. Or maybe I have it backwards and linq/transducers really are partial differential equations and C# snuck it into the 4th grade curriculum and nobody noticed.

javajosh 3y ago

I put together a transducer in JavaScript with the intent to simplify Redux/flux architected front-ends. Rather than have synthetic actions that trigger reduction, I allow the caller to simply add objects directly into the store, and rely on them being combined in a useful way. And rather than leave it purely at dead data, I allow you to add functions, too, such that once added they can handle subsequent input. My `combine()` is a recursive `Object.assign()` plus reification of function calls: https://simpatico.io/combine2 https://simpatico.io/combine2. It's that second part that makes it a transducer. This work predates the term "transducer" and I was happy to see Rich Hickey defining it, so I use the term retroactively. Note that I don't see much use for transducers as a general programming technique. This one use is very specific, but I have never before or since needed one.

vanderZwan 3y ago
> This work predates the term "transducer" and I was happy to see Rich Hickey defining it IIRC Hickey did not define this term (nor claimed to have done so), he found it in existing compsci research papers that approached them from a mathematical proof angle and realized they solved a problem he needed to solve (basically not allocating intermediate collections and doing wasteful work on parts of the data that will be immediately discarded). I think one of the authors of the papers he cited even gave a talk at a Clojure conference once.
eyelidlessness 3y ago
FYI: I was curious and clicked your link, but it fully reloads the page every 2-3 seconds so I had to bail (iOS Safari if that helps)
- javajosh 3y ago
  Thanks for the feedback, fixed. I like to use a simple method to speed up my BTD loop when doing webapp work, which is to add a `meta refresh` tag. But that's inappropriate for deployment. Note that my site is very strict about 3rd party content (none is allowed), and about privacy (the only log is in a tmux buffer), so there's no reason for this behavior other than my forgetfulness.

sesm 3y ago

Initial problem was duplication of sequence library for CSP channels (core.async in Clojure). When the idea of ‘transducers’ (transformers of reducing function) was discovered, it turned out to have many additional useful properties and was integrated deeply into standard library.

thom 3y ago
I’ve always felt any mention of reducers is a big distraction. They’re not really key to the idea of transducers which are almost entirely about pure sequence to sequence transforms.
- thom 3y ago
  For the benefit of downvoters, let me explain myself: the 'reducing function' (also called a 'reducing step function' in other parts of the docs) isn't always what you'd think of as the normal arity-2 function passed to reduce itself. Take a look at the reducing functions in clojure.core - the 3-arity, innermost function returned inside all the transducers. They almost never do anything with result which would be the accumulator in a normal reduction - they just pass it on or return it. In fact, it wouldn't make much sense for transducers to do any sort of reduction because if they did, you'd no longer be doing a step-wise transform over a sequence, and you could just use normal function composition because you'd have a single value to pass forward. The reduced mechanism is only really used in transducers to short circuit, so even there the name clouds the meaning. The only time that something we'd recognise as reduction is happening is when you call transduce and specify a reducing function as its f argument. You'll have seen lots of examples with + as the reducing function because it handily returns 0 when called with no args, acts as identity with 1 arg, and reduces (actually reduces!) with two args. But instead look at sequence which can take an xform. It's implemented in Java code, but the inner reducing function it uses completely ignores the accumulator value! Everywhere you're using result inside a reducing function? It's just null and ignored. But sequence elsewhere stashes away the (transformed) output of the transducer stack and lets you have it back as a sequence. No reducing is happening, just transforming sequences. Elsewhere, into supports an xform argument - what's its reducing function? It's just conjing stuff back together! So to me, the interesting bits of transducers are that they give you an efficient and composable way of creating sequence-to-sequence transformations. Any and all mentions of reducers are necessary plumbing but slightly distracting from the core ideas.