4 ms·
Okay, so you know streams in Java. You know a Collector? A transducer is roughly a Function<Collector, Collector> (with some more type variables to make that va
by twic 3y ago
Okay, so you know streams in Java. You know a Collector? A transducer is roughly a Function<Collector, Collector> (with some more type variables to make that valid). Given a Collector, a transducer will make another Collector which does some stuff to its input and delegates to the underlying Collector.
The nice thing about this is that you can implement all the non-terminal operations on Stream in terms of this. Mapping is passing each input element through a function before passing it on. Filtering is testing each element with a predicate, and only passing it on if it passes. Flat mapping is passing the element through a function to get a number of new elements, then passing them all on one by one.
But you can do more! You can write a transducer which batches elements, a bit like the inverse of flat mapping. You can't do that with the existing Streams API at all!
The things transducers operate on, which Clojure calls reducing functions, end up being a bit different to Collectors: their accumulate operation returns a boolean to control early stopping, which lets them implement take while, limit, and so on, and that means they can't be parallel. But the vibes are exactly the same.
- tadfisher 3y agoA similar mechanism in another JVM language is Kotlin's sequences, which can also do windowed/chunked/batched operations by holding a little bit of intermediate state. Combine that with coroutines and it's pretty trivial to perform a parallel map/flatmap and saturate all cores. I'm surprised Streams can't do this.
- twic 3y agoIn case what i wrote wasnt clear: Java's streams are (or can be) parallel, and can saturate as many cores as you like. But parallelism is not compatible with early stopping at any stage - early stopping is inherently serial! - and transducers choose allowing any stage to stop early over being parallel.
- dataangel 3y agoThis still just sounds like boring old iterators. Rust and C++ iterators both do all of this.