7 ms·
Transducers for Python
- tallerholler 11y agoeli5: what can you use these for? what are some examples of when you'd use something like this...
- fnordsensei 11y agoThe idea is that you can define a transformation separate from its use and then reuse it in many contexts. Some of the win sits in decoupling what you are doing from when and where you are doing it. Rich Hickey wrote about them on Cognitect's blog a while back: http://blog.cognitect.com/blog/2014/8/6/transducers-are-coming http://blog.cognitect.com/blog/2014/8/6/transducers-are-comi... The author refers to the test suite for examples: https://github.com/cognitect-labs/transducers-python/blob/master/tests/transducer_tests.py https://github.com/cognitect-labs/transducers-python/blob/ma...
- ufo 11y agoI always had a hard time figuring out how transducers are different "stream transformation" functions like map and filter. I know that they are more complicated/powerful than that but I can't quite point that out precisely.
- deleted 11y ago[deleted]
- fnordsensei 11y agoYou can transform map or filter into a transducer by decoupling the transformation from the data. When you call map, you usually call it on some data, or some stream. When you create a transducer, and you call for example (map inc), you're saying "whenever this transducer is applied, on whatever it is applied to, increase each unit by one". You can lug around the transformation itself as if it were an object, almost, separate from the data it's meant to transform.
- ufo 11y agoHow is that different from just partially applying/currying the map?
- closed 11y agoA key component to transducers is to realize that map, filter, take, can all be expressed as special cases of the reduce function. So a curried map function can be expressed as a reduction operation, for example, in python you could do.. # map that adds 1 to each element # same as curried map(sub1)([1,2,3]) sub1 = lambda x: x - 1 map(sub1, [1,2,3]) # same outcome using reduce reduce(lambda acc, x: acc + [sub1(x)], [1,2,3], []) Notice that when writing a map operation using the reduce function, we've put two pieces of logic on each step. (1) we have a transformation called sub1, and (2) we have a step operation (returning an array with the transformed x appended to it). We can use this same logic to create filters with an if statement.. lt2 = lambda x: x < 2 filter(lt2, [1,2,3]) reduce(lambda acc, x: acc + [x] if lt2(x) else acc, [1,2,3], []) A critical thing to notice in each of these cases is that if we ran a curried filter and map (with transformation / filtering functions already passed), then they would loop over the data twice. However, if you look at the example on this github repo, the want to * take 3 elements from the sequence * perform a transformation (map operation) to those elements Notice that in this case take is being performed before the map, but you could imagine doing something like * map items from geometric series to float * filter items that are greater than .15 * take 3 of the remaining items How would you do that with map and filter functions without looping over the entire (inexhaustible) sequence? This is the beauty of transformers and transducers, they provide a composable approach to filtering, transforming, and taking elements from a (possibly infinite) sequence. Of course, all of this could be done in a for loop, but often it introduces complexity.. # assume geom_series is the generator from the github example take = 3 for el in geom_series: if float(el) < .15: result.append(el) if len(result) == take: break However, this for loop is using the same ideas: we have a transformation / if (predicate) operation, and step operation to append the data we want. The problem transducers try to solve is writing a series of these types of operations compactly and cleary as a pipeline of functions (erm, transformers), so those functions can be replaced/moved around quickly, and the pipelines can be combined together simply. A good article that covers tranducers in javascript (sorry!) is here: http://simplectic.com/blog/2014/transducers-explained-1/ http://simplectic.com/blog/2014/transducers-explained-1/
- deleted 11y ago[deleted]
- fermigier 11y agoA similar, but different, project, with associated learning material: - https://github.com/abingham/python-transducers https://github.com/abingham/python-transducers - http://www.slideshare.net/alinadolgikh/austin-bingham-transducers-in-python http://www.slideshare.net/alinadolgikh/austin-bingham-transd... - http://sixty-north.com/blog/series/understanding-transducers-through-python http://sixty-north.com/blog/series/understanding-transducers... The last link is an 8 part blog series discussing transducers from first principles, and discussing their use (still in the context of Python).
- rob_smallshire 11y agoNote that the first link has been superseded by https://github.com/sixty-north/python-transducers https://github.com/sixty-north/python-transducers which is also available on PyPI as https://pypi.python.org/pypi/transducer/ https://pypi.python.org/pypi/transducer/
- danidiaz 11y agoMy own attempt at implementing something like transducers in Haskell: http://hackage.haskell.org/package/foldl-transduce http://hackage.haskell.org/package/foldl-transduce. It depends on the useful "foldl" library of reducers.
- boothead 11y agoThis looks really useful! Isn't your Transduction the same as doing premap (or lmap from Profunctor)?
- danidiaz 11y agoTransformations created with premap are stateless, they depend only on the current input. So one can't build UTF8 decoders with them, for example. That said, if you supply a function to premap, the result typechecks as a "Transduction".
- rob_smallshire 11y agoThis code by Cognitect is a fairly literal port of Clojure transducers to Python. If you'd like to see a more Pythonic interpretation of transducers, take a look at Sixty-North's alternative https://github.com/sixty-north/python-transducers https://github.com/sixty-north/python-transducers
- agumonkey 11y agoAlso transducers for Common Lisp https://www.reddit.com/r/Clojure/comments/3tha86/transducers_in_1989/ https://www.reddit.com/r/Clojure/comments/3tha86/transducers...
- fulafel 11y agoThe README sounds like it's describing Python's venerable iterator protocol. The example code looks like higher order functions and iterator composers from the standard functools and itertools modules. It should probably explain how this is different from those if the audience is Python programmers.
- anon4 11y agoI concur. I don't really get the name "transducer". It's just map+reduce? So... why do we need a new name and new libraries?
- tel 11y agoIt's not really map+reduce. Transducers arise out of realizing that transformations between (possibly infinite) lists can be encoded as a "reducer transformations", "transducers". When you work in this encoding you automatically get lazy transformation (since everything goes through a function call) and the avoidance of intermediate data structures (since everything goes through a function call!). Moreover, there's a neat property of this encoding, a CPS encoding, which is that function composition still "works", it just works "backwards".
- fulafel 11y agoPython iterators also give you lazy transformation and no intermediate data structures, no?
- anentropic 11y agoGenuine question: what is the point of this? it seems like a long-winded and not very clear way to write something you could just do with a function and a list comprehension
- moomin 11y agoWell, in Clojure they solve the problem that the standard functions are inefficient when you chain a bunch of list manipulations together (because they all return lazy lists) and neatly sidestep the fact that Clojure doesn't have generators.
- ch 11y agoSo if Clojure had some form of fusion of composed functions on lists, that would equate out to what transducers are doing?
- fnordsensei 11y agoThat's almost literally what happens. (comp (map inc) (filter even?)) when applied takes all even numbers from a collection/channel/what-have-you and adds one to them. Filter and map normally take a collection as their last argument, but when none are supplied, a transducer is returned instead.
- deleted 11y ago[deleted]
- espeed 11y agoNB: Here's the Clojure page for transducers, which includes a link to Rich Hickey's introductory blog post and Strange Loop video. http://clojure.org/transducers http://clojure.org/transducers
- amirouche 11y agoHow does transducers compare to Gremlin traversal language?