8 ms·
Comprehensions in Python the Jedi way
- vadim41 11y agoCute oneliners that are close to unreadable and definitely unmaintainable. If I see this kind of "cute" code in code reviews, there is some serious scolding to be done.
- Niksko 11y agoThat seems a little harsh. Nested list comprehensions with multiple filtering if statements? Scald away. But the simpler examples there are perfectly readable, understandable and maintainable if you understand list comprehensions. And if you need to alter it to the point where it needs to be a little more verbose in order to convey what you're doing, then sure, you can break it out into some other structure. But just because the complexity might increase later on doesn't mean that you shouldn't use a list comprehension.
- matt_wulfeck 11y agoBut what do you gain from using nested list comprehension like that? There's no performance gain (or very, very little). The only tangible benefit I see is saving a few lines of code. And for those few lines of code you've traded the ability for new programmers to understand it easily. To me that's a net loss.
- spamizbad 11y agofor-ins leak iteration variables into the function scope, which can introduce subtle bugs if you're not careful. set/dict/generator comprehensions do not. Example: for foo in bar: baz(foo) print foo # Will actually print something Whereas whatever = {baz(foo) for foo in bar} print foo # NameError: name 'foo' is not defined does not put foo into the outer scope. list comprehensions in python 2 (but not 3) will leak variables however.
- orf 11y agoThere is always a performance gain, and sometimes it can be fairly huge.
- Retra 11y agoNew programmers should learn how to use list comprehensions. They're not just some obscure Python concept, they appear in multiple other languages, as well as mathematics.
- RodericDay 11y agoIt's actually much more sensible code a lot of the time, especially for people who didn't get a degree that consisted of a lot of for-looping in class. good_jedis = [jedi for jedi in universe if jedi.alignment == "good"] vs. good_jedis = [] for jedi in universe: if jedi.alignment == "good": good_jedis.append(jedi) And then when you get on stuff like dictionary comprehensions and generator expressions, it starts becoming an amazing tool that is able to accomplish things that are straight-up cumbersome with old loops. There's a computer science course out there that a friend was telling me about, that begins by showing students how to "map" a function over an iterable, way way before a "for loop" ever shows up. I think this is the way it should be done, "for loops" as a three-line-structure only seem natural to people who grew up programming for loops.
- golergka 11y agoI just learned about this, so they are unreadable to me. But it's not relevant. What relevant is, do they create a readability problem for a python developer who spent significant time with them? I honestly don't know.
- andybak 11y agoI 'got' list comprehensions fairly immediately and find them clearer in most cases than map/filter for an explicit loop. I'm cautious when nesting them (as long as your code formatting is clear - a single nested comprehension is pretty acceptable) and I never use the multiple 'for' form as it's just not intuitive to me. I am rather fond of dictionary comprehensions as mentioned in the article: colors = [jedi['lightsaber_color'] for jedi in jedis] frequencies = {color: colors.count(color) for color in set(colors)} print(frequencies) # {'green': 6, 'red': 5, 'blue': 6}
- bpicolo 11y agoI think the biggest reason python map/filter aren't natural is the building outwards (though I do realize it's lispy and some people might prefer it). Languages that support it directly on a list are much more intuitive in my head: list.filter().map()... Alternatively, pull in syntax like elixir (and other languages) have to make chaining a breeze: list |> map() |> filter()
- ardfard 11y agoReally? I am pretty much mind-blown when the first time I learned about this. Given that my background is Mathematics, I thought it was pretty elegant and also looked very intuitive.
- deleted 11y ago[deleted]
- spamizbad 11y agoEh, while comprehensions with multiple for-ins push the limits of good taste, I don't think: planets_set = { planet for episode in episodes.values() for planet in episode['planets'] } is less maintainable in a Python shop than say... planets = set() for episode in episodes.values(): planets.update(episode['planets']) Although the latter will likely make perfect sense to most non-python developers. The former is faster and has a smaller memory footprint and might be preferred when dealing with a larger or more irregular data sets. The former also has the advantage of not leaking the "episode" variable into the function/method scope, which could introduce a subtle bug if that variable gets conditionally reused. So while it's harder to understand for a less-experienced python developer, the set comprehension solution is inherently safer due to python's design.
- bpicolo 11y agoThough in py2 list comprehensions do leak scope. (dict comprehensions don't, you are correct on that :).
- AnkhMorporkian 11y agoAdditionally, generator comprehensions can be far more efficient than any simple for loop. If you had a generator with a billion star coordinates being read from some file, you'd never be able to load it all into memory. So, instead of manually making a generator function, you could just do coordinates = (star.x, star.y, star.z for star in star_map)
- bpicolo 11y agoList comprehensions are both insanely readable and easy to maintain. They're one of the things python gets very, very right
- rcarmo 11y agoActually, these are pretty idiomatic and understandable. List comprehensions (and inline generators, which you get by using parenthesis) make it trivial to handle fairly complex sequence processing, and are one of the reasons Python gets a lot of love from the FP community. Sure, it's not really an FP language, but these give it an almost LISPy feel. Regardless,it would be less Pythonic to build a loop and iterate. Might be easier for complete novices, but I don't suppose people are just hiring novices these days, right? (edit: Mobile-sponsored typo)
- andybak 11y agoGosh. If I was told off for using list comprehensions I'd start considering my employment options fairly quickly. Whilst they can be abused and pushed beyond the limits of reasonable use - in most cases I've seen in the wild they are the clearest and most Pythonic way to do it.
- jqm 11y agoThat's not really "cute" code though. Short list comprehension isn't any less readable (and certainly not less maintainable WTF?) than a multi-line "for" loop. In fact, I find short list comprehension more readable. I don't have to mentally keep track of place in a multistep loop.
- kelseyfrancis 11y agoI think I'd resign if someone in a position to be reviewing Python code scolded me for using a list/dict/set/generator comprehension.
- a_bonobo 11y agoOne interesting side-thing with list comprehensions in Python 2 vs. Python 3: Python 2.7: >list_of_numbers = [1,2,3] >[x/2 for x in list_of_numbers] >print(x) 3 Python 3: >list_of_numbers = [1,2,3] >[x/2 for x in list_of_numbers] >print(x) NameError: name 'x' is not defined They "leak" their variables in Python 2, if you're someone who reuses variables this can lead to an enormous headache!
- Cyph0n 11y agoWow, I never noticed that! Who the hell thought that was a good idea?
- bpicolo 11y agoPretty sure it was a bug that they kept for backwards compat, not intentional, though I may be wrong. Py3 let them shed it off though.
- masklinn 11y ago> Pretty sure it was a bug that they kept for backwards compat, not intentional Yep, as demonstrated by only list comprehensions exposing this behaviour (on python 2) whereas none of generator comprehensions, dictionary comprehensions or set comprehensions do: >>> [x for x in range(5)] [0, 1, 2, 3, 4] >>> x 4 >>> {y for y in range(5)} set([0, 1, 2, 3, 4]) >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>> {y:y for y in range(5)} {0: 0, 1: 1, 2: 2, 3: 3, 4: 4} >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>> list(y for y in range(5)) [0, 1, 2, 3, 4] >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>>
- bpicolo 11y agoTo be fair, generators are lazily evaluated so I would sure hope they wouldn't expose scope! That would show up in truly unique places haha
- d0mine 11y agoThere is an easier way to convert bits to ASCII, filter spaces and reverse the string: >>> n = int(bbs, 2) >>> n.to_bytes((n.bit_length() + 7) // 8, 'big').decode() 's no i sn e h e r pm o c' >>> _.replace(' ', '')[::-1] 'comprehensions' http://stackoverflow.com/questions/7396849/convert-binary-to-ascii-and-vice-versa http://stackoverflow.com/questions/7396849/convert-binary-to...
- bearfrieze 11y agoI like the replace method. It's a great way of doing the same thing. I considered using the [::-1] syntax to reverse the list, but decided that there was enough "cute" stuff in the examples already.
- d0mine 11y ago[::-1] is an idiomatic way to reverse a string in Python that is the obvious way for a habitual user of the language to do it e.g.: def palindrome(s): return s == s[::-1] It could be discussed whether ''.join(reversed(s)) is more readable for a novice programmer learning Python. In general, Python prefers words over punctuation. Also, there are objects that can be reversed() that are not sequences. http://stackoverflow.com/questions/931092/reverse-a-string-in-python http://stackoverflow.com/questions/931092/reverse-a-string-i...
- tyingq 11y agoAnother alternative, works on both py2/3, assuming you replace bbs='000... with bbs=b'000... import struct result=''.join([chr(int(bits,2)) for bits in struct.unpack('8p'*(int(len(bbs)/8)),bbs)]) print(result.replace(' ','')[::-1])
- ambicapter 11y ago> planets_flat = [planet for episode in episodes.values() for planet in episode['planets']] Can somebody explain this one to me (I understand list comprehensions)? I'm having trouble understanding how the second part uses something defined in the first part, but the first part can't stand on its own, so > [planet for episode in episodes.values()] returns an error.
- thomasahle 11y agoIt's equivalent to planets_flat = [] for episode in episodes.values(): for planet in episode['planets']: plants_flat.append(planet) Notice how the for loops in the comprehension goes in the same order as in the imperative code.
- hobarrera 11y ago> Notice how the for loops in the comprehension goes in the same order as in the imperative code. Thanks, that's a sane way to explain the order. Up to right now, it was always "the opposite of what you'd expect", which was a memory rule that always failed me.
- thomasahle 11y agoIt also works with the if-filtering, which usually goes in the inner most loop. Actually I don't recall if you can put an if between two for's, like you might do in an imperative loop.
- thomasahle 11y agoIt also helps on how ifs should be inserted, for example: ys = [] for x in xs: if P(x): for y in Y(x): if Q(x,y): ys.append(y) Becomes [y for x in xs if P(x) for y in Y(x) if Q(x,y) ] Surely combining this many for/if's may often be the wrong idea. Just like making a depth 4 iterative loop isn't always ideal. It does make the order easier to remember though :)
- craigds 11y ago
- bpicolo 11y agoList comprehensions are awesome. Not only that - python does them insanely beautifully. Clojure and ES6 are examples that I think aren't as readable, though equally powerful more or less. But in python they don't have any clunkiness. Simple, expressive. Love them. Nesting them can get ugly, but it's easy to avoid: Just use generator expressions and chain them. No real runtime overhead that way.
- nilliams 11y agoWell with regards to clunkiness and awesomeness, are they not still less 'naturally' composable and (as a result) less readable in composition than collection pipelines? I don't see a good reason to prefer them over: collection .map(x => x * 2) ... .filter(isOdd) .reduce(blargh) ... style syntax that most other modern, C-style languages offer now (ES6, Rust, Ruby, C# ...)
- bpicolo 11y agoI think there are values in both styles. I've written a lot of both and for simpler things (which the vast majority if not all of code should be), I definitely prefer list comprehensions. They feel even more functional because they don't depend on map/filter/reduce defined on your objects. I think chaining syntax from e.g. Elixir gives you more flexibility than map/filter/reduce defined on objects too. Big fan of that, though it does interact oddly with elixirs optional parentheses for function calls (which is a mistake of the language imo)
- thomasahle 11y agoThe comprehension approach, while requiring more syntax, seems to often produce a more 'natural' order of operations. For example, compare [p for p in range(2,100) if all(p%q!=0 for q in range(2,p))] with range(2,100).filter(p => range(2,p).map(q => p%q!=0).all()) It might be a small thing, but in the first one I feel the prime 'p' becomes the center piece, whereas in the second, 'p' is burrowed somewhat in the expression.
- deleted 11y ago
- undershirt 11y agoI don't think the author realizes how appropriate the Jedi tone for this article is. Comprehensions are the gateway drug to the dark side, away from imperative programming and toward languages that treat everything as expressions which snap together more freely. It hints at an idea of making a `for` loop and an `if` statement return a value (see CoffeeScript). But it also hints at the idea that useful idioms like the List Comprehension can merge/simplify existing constructs into a new, easier syntax-- and that there are languages that allow you to do this freely (see Lisp). Python showed me the Force, but I'm with the Dark Side now.
- seivan 11y agoSorta like signals using RxJS/RxSwift.
- hellofunk 11y agoOne of the most extraordinary APIs for comprehensions can be found in Clojure, where the "for" expression really changed how I view data. A single expression can generate many complex data sequences with such elegant beauty.
- bearfrieze 11y agoAuthor here. Thanks for this brilliant comment. May the force be with you.
- rebootthesystem 11y agoList comprehension in Python is great. However, that's Padawan territory. If you want to be a Jedi then APL is the only way. It's not "list comprehension" it's the way of the force when you use APL.
- MatthewWilkes 11y agois should not be used to check string equality, as it is in the space filter. This only works as CPython interns short strings automatically.
- bearfrieze 11y agoThanks for pointing this out along with some folks over in the comments on GitHub. I've updated the Gist.
- cgriswald 11y agoPedantic SW points: 1. Rey is not a Jedi. A better dict key would be "fav_force_user". 2. For the planets, Episode II is missing Coruscant, Episode VI is missing Dagobah. Edit: formatting
- jordigh 11y agoHah, like there is any doubt that she'll become a Jedi. It's basically Episode IV all over again. Han Solo instead of Obi Wan, Rey instead of Luke, First Order instead of the Empire, Kylo Ren instead of Darth Vader. We like hearing a story we already know. Rey is well on her way to become a Jedi, probably in two-movies' time.
- zo1 11y agoAnd don't forget, old Luke is probably going to act as the Yoda-equivalent in the next "episode".
- bliti 11y agoThanks, I always get my trek wars trivia mixed. Which movie is the one where Chewbacca goes into time and rescues some whales? //Just some Sunday fun. :)
- bearfrieze 11y ago1: Seems pretty clear that The Force Awakens is an "...origin story of a female Jedi." [1] 2: At some point while writing this I realised I would spend all evening if I had to round up all the planets, and chose to note that the lists are "non-exhaustive" instead :) [1]: http://www.wga.org/content/default.aspx?id=6130 http://www.wga.org/content/default.aspx?id=6130
- matt_wulfeck 11y agoI don't want to be a jedi Python programmer and exploit every neat trick of the language. I want to be a very good programmer and write code that's easy for others to read, debug, and maintain.
- santaclaus 11y agoI wouldn't call list comprehensions a Jedi feature of Python -- they are pretty darn idiomatic and common.
- JustSomeNobody 11y agoComprehensions are not a neat trick. If you know and use Python, writing comprehensions is writing good code that can be understood by others.
- unoti 11y agoComprehensions are great, but they can hurt readability and maintainability when taken too far. Most of the time you shouldn't be playing code golf with your code, iteratively seeking to pack more and more work into a single line of code. That makes the code harder to understand and harder to re-use. While many of the examples in this helpful article are good, the first example with the octets is an excellent example of how not to do it. Look at the octet parsing code we ended up with in the article: # Snippet 1 octets = [bbs[i:i+8] for i in range(0, len(bbs), 8)] It's nice and tight. What does it do? I'd need to peer at it a moment and decode it, executing it in my head. This is subjective, but I think code should be self-explanatory; it's up to the computer to execute code, not people in their heads. Is there an off-by-one error in there? Here's another way to do it that'd be better. octets = chunks(bbs, 8) That function chunks() is something that I keep in an iterutils package which I end up using all the time. It's intuitive, and it has a doctest that shows that we definitely don't have an off-by-one error. It's also easier to re-use than the first one. Maybe it also bears mentioning that chunks() works on an iterator, while the first solution needs to keep the whole thing in memory at once. Here's the chunks method I use: def chunks(collection, chunk_size): """Divides list l into chunks of up to n elements each. >>> l = range(75) >>> chunks(l,10) [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], [20, 21, 22, 23, 24, 25, 26, 27, 28, 29], [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], [40, 41, 42, 43, 44, 45, 46, 47, 48, 49], [50, 51, 52, 53, 54, 55, 56, 57, 58, 59], [60, 61, 62, 63, 64, 65, 66, 67, 68, 69], [70, 71, 72, 73, 74]] """ for i in xrange(0, len(collection), chunk_size): yield collection[i : i + chunk_size] When you catch yourself playing code golf and trying to pack more and more meaning into a single line, look for ways that you can break the problem down into multiple components that use each other. This kind of functional decomposition is one of the things that makes functional programming so wonderful. Lots of times the intermediate steps in a complex expression have meaning and are useful on their own.
- kilburn 11y agoI totally agree with you on the core issue. However, I would reject your `chunks` function in a code review and tell you to use `grouper` form the itertools recipes [1]. More generally,any time I've ended up with a long list comprehension, the answer has been to "check itertools and see how you would describe this ugly comprehension in those terms" [1] https://docs.python.org/3/library/itertools.html#itertools-recipes https://docs.python.org/3/library/itertools.html#itertools-r...
- RubyPinch 11y agoI just wish that python could stop being such a butt about not being like the rest of the languages list of items --> filter list --> operate on list becomes operate on list <-- (list of items --> filter list) And because this is how it was decided to tackle map/filter problems, we'll always have a weird gimped anon-function operator instead, to discourage the map/filter patterns of every other language
- distracteddev90 11y agoOne thing I find interesting is that everyone loves to hate CoffeeScript, but its individual features/syntax are consistently lauded in conversations about other languages. (Not to mention half of ES6 existed in CoffeeScript first, but that's a gripe for another day)
- stuartaxelowen 11y agoThe composition of features is more important than just the individual features - CoffeeScript is a great example of when feature composition goes wrong.
- catnaroek 11y agoList comprehensions are just syntactic sugar over some common higher-order functions. Are we really getting excited over syntactic sugar? How about making Python's higher-order functions not suck instead?
- alexandercrohde 11y agoOr, there's a 1-million times better way to do this with a good library. In javascript for example FA('0...11'.split('')).chunk(8) .map(x => parseInt(x.join(''), 2)) .map(x=>String.fromCharCode(x)) .filter(x=>x!=' ') .reverse() https://github.com/anfurny/Fancy https://github.com/anfurny/Fancy
- Negative1 11y agoIt's fun switching from Python to Scala where for/yield comprehensions are idiomatic and very natural (nested comprehensions being a good example). Oh, and type safe and fast. It's especially powerful to build off those constructs with currying, partial application, pattern matching, etc... I love Python and write code in it daily but as a (pseudo-)functional language it feels very awkward to me.
- jonesb6 11y agoYes let's make the reputation of Python more cryptic and culty. Because cryptic and culty things are better right? Han Solo said it best "Hokey religions and ancient weapons are no match for a good blaster at your side, kid." Lets keep it explicit alright? It's better then implicit. Edit: read the article, it neither makes python cryptic or culty. The Jedi thing is just a cool SW reference. That said list comprehensions are a little cryptic to me since I haven't written python in awhile. Personally I think an important attribute of elegance in programming is how little you need to read the docs to understand something, and the more one-liners we do the more times we are likely to have to look at the documentation before reading it (not necessarily a bad thing, but sometimes a time sink and sometimes people won't look up the docs when they should!).