10 ms·
Brilliant or insane code?
- hartror 13y agoThis is in the zip documentation as the way of solving this problem. Sort of surprised the author didn't look up the documentation before writing what is otherwise a very good post. The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups using zip(*[iter(s)]*n). http://docs.python.org/2/library/functions.html#zip http://docs.python.org/2/library/functions.html#zip
- StavrosK 13y agoI am also surprised I didn't look it up. I guess that I didn't expect that anything more than how zip() works would be there, and I already knew that. I've updated the post with this, another commenter pointed it out. Thanks!
- akavel 13y agoBy the way, I was recently surprised by a similar pattern in linux, where you can do: (echo -e "one\ntwo\nthree\nfour") | paste -d, - - to get result of: one,two three,four by exploiting a similar trick, i.e. reading two times ('- -') from the same iterator (STDIN of 'paste')
- apw 13y agoThanks for posting this. For reference, paste is a standard UNIX utility, whose purpose is to do all sorts of useful things not limited to 2-tuples: $ seq 1 9 | paste - - - 1 2 3 4 5 6 7 8 9
- tambourine_man 13y agoNo need for parentheses, echo -e "one\ntwo\nthree\nfour" | paste -d, - - works just fine
- impressive 13y agoRecently I was downvoted 2-3 times on S.O. for an answer that was claimed to be non-idiomatic. So, I cleaned it up, but it really irked me. What I had written was totally fine. It shouldn't have hurt anyone's eyes. It was direct. It was in-your-face. It was not magic. Reading this post brought back that feeling. If people don't understand a completely valid and terse way of coding something, sometimes instead of bothering to understand it, they will bash it. Sometimes, this is a totally valid way to vent frustration, and then they learn something new, and all is good. But sometimes, it just gets left as "this is wrong" and then someone else thinks it is wrong, and so on. That is wrong, and tech leads or architects that enforce such crap will bug the living shit out of good developers and lose them. I appreciate clarity. But, terse one-liners can be just as clear if not clearer than code that unnecessarily adds more methods/functions/names/local vars and claims to be "more testable", etc. You shouldn't have to sacrifice the ability to be terse and clear at the same time. Testing is no excuse for code bloat. You can likely write a test that executes the behavior without having to atomize it. Assess the amount of production and test code you are writing. How much more code are you actually having to write in order to test, both in the tests themselves and in the code which you are having to test? It happened here with Python and it happens in many languages. It even happens with laws and regulations in government. If someone gets the same thing done just as ethically but without the bureaucracy, just appreciate it as another perhaps better way of doing something. Don't bash it publically because you don't understand it.
- chipsy 13y agoI think the mentality can be extended to most code-reading. Don't get it by skimming? Must be crap code. I only got away from that when I changed my litmus test towards whether I could write on top of the codebase successfully, not how it looked. Today my only real point of judgment about the look of code is whether it's written in a style that increases average error rate. w/r to Python in particular, it has a history of ending up with idioms that are "tricky" and not particularly more or less terse than other techniques, but are able to exploit the standard library functions to get a faster-running result. This is, of course, at odds with the motto of "there should be only one (obvious) way to do it," so every experienced Python programmer has to internalize a small dictionary of idiomatic one-liners for these exceptional cases. (Fortunately, it's not that big. I can only think of three or four off the top of my head.)
- norswap 13y agoKind of ironic for a language whose motto is "explicit is better".
- StavrosK 13y agoIt does say "possible", not "recommended" or even "sane"...
- catwell 13y agoThe fact that this idiom is documented in the official Python documentation makes me cringe, knowing that this code is not what I would call explicit. This is one of those cases where Ruby does it better (s.each_slice(3).to_a).
- rational_indian 13y agoBrilliant and insane?
- StavrosK 13y agoHah, yep, I hadn't considered it can be both.
- lmm 13y agoInsane, because it relies on the zip implementation detail. If you cared about a measly factor of 4 in performance you wouldn't be using python anyway.
- icebraining 13y agoIt's not an implementation detail, the order is guaranteed by the spec.
- Goopplesoft 13y ago> by the spec by the implementation you mean. See: http://stackoverflow.com/questions/1094961/is-there-a-python-language-specification http://stackoverflow.com/questions/1094961/is-there-a-python...
- nsmartt 13y agoAs mentioned by others, the documentation for zip explicitly mentions this as a Python idiom: http://docs.python.org/2/library/functions.html#zip http://docs.python.org/2/library/functions.html#zip
- techdragon 13y agoAny developer can care about performance. And should never be mocked for achieving above 200% performance compared to an alternative implementation... Let alone 400% performance compared to the alternatives.
- lmm 13y agoIf that 400% performance improvement was achieved by sacrificing code readability, on a project that has deliberately chosen to prioritize readability over performance, then yes they should be mocked. Or at least told not to do it.
- signed0 13y agoIt's not brilliant. This accomplishes the same thing without being hard to understand: from itertools import islice iterator = iter(array) try: while True: yield list(islice(iterator, 3)) except StopIteration: pass
- StavrosK 13y agoSure, there are many ways to do it, but I think the author was going for speed here.
- signed0 13y agoI'd be surprised if my way is slower. Any time you unpack into a function such as zip() python has to create an intermediary list to store all the results before calling the function.
- merlincorey 13y agoExactly, I would be very surprised if the zip version was faster. One of the first steps to optimization in python is moving everything you can to generators and using of itertools. The OP's question of is this genius or bad is clear in that regard: it is bad, due to not being the proper optimization direction, but it is interesting.
- StavrosK 13y agoI would be very surprised if it weren't. The zip version has the hot path written in C, the other version has it written in Python with a bunch of exception handling/list assignment code, etc. The original method is just zip(iterator, iterator, iterator), which is probably as fast as anything can be.
- elyase 13y agoThat is a common misconception. Moving to iterators adds a function call while list creation in C is quite fast. Every case has to be tested for performance.
- bjourne 13y agoI think the code is pretty ok, except for the stupid name, docstring and that it is a method and not a free function. def chunks(seq, n): "groups the elements of the seq into a list of n-sized chunks." return zip(*[iter(seq)]*n)
- inglesp 13y agoThe original docstring is a complete red herring... yours is much better, but I'd go one further and include an example.
- mjburgess 13y agoIt does not rely on an implementation detail, that is how iterators work. He's just supplied the same iterator to a function which consumes iterators... that's exactly the expected behaviour.
- tgb 13y agoIn fact official python docs say: "The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups using zip([iter(s)]n)." http://docs.python.org/2/library/functions.html#zip http://docs.python.org/2/library/functions.html#zip
- tiziano88 13y agoThe fact that zip evaluates its arguments in order is an implementation detail. It could evaluate them in reverse order, in which case this code would not behave as expected.
- StavrosK 13y agoThat's what I meant, thanks for clarifying. It does seem that the zip() docs make a guarantee of left-to-right evaluation, though, so my apprehension proved unfounded.
- hknozcan 13y agoWe all love short and fast. But this is definitely an interesting approach. I'd love to see similar approaches to problems if you guys can point out to some.
- StavrosK 13y agoI would also, I love this sort of thing. How about: >>> some_boolean = False >>> ["Thing 1", "Thing 2"][some_boolean] "Thing 1"
- icebraining 13y agoYeah, but unlike the hack in the article, the more readable version of that code, "Thing 1" if some_boolean else "Thing 2" is also almost twice as fast(775 vs 1340 ns, on my machine).
- StavrosK 13y agoI was more posting it for the "cute hack" value, rather than speed. Speed was significant in the post because the author needed their library to be fast.
- _ZeD_ 13y agothe problem with this expression is that it doesn't "short circuit". Both "Thing 1" and "Thing 2" (in this cases object, but the can be function calls) are evaluated before the "some_boolean" usage.
- TheLoneWolfling 13y agoThat can be an advantage in some cases, however. Not generally, though. In particular, if you want the side effects of both operations. Trying to write multiple things while checking if any of them failed, for instance.
- hknozcan 13y agoShort but I doubt this will be faster as noted.
- scotty79 13y agoi = iter(array) return zip(i, i, i) There you go. All but neceessary magic gone with just one line more.
- StavrosK 13y agoMuch, much better.
- hartror 13y agoA much clearer solution, I would also add a comment reminding people how zip works.
- artursapek 13y agoReally? What's the point in explaining a standard language function that they should know, and can just Google if they don't remember?
- ekimekim 13y agoIt's using a somewhat-obscure guarantee that doesn't come up in normal usage of the function - namely, that it will always get the iterator values in left-right order.
- andrewcooke 13y agoit depends on the kind of code you write. i guess if you're writing web server stuff, documenting this makes sense. but in maths-related code, it's pretty standard. you use something very similar to transpose matrices, for example. (and the original article is dealing with coords in graphics, which is "maths-related code" in my book, but perhaps not in everyone's)
- loarake 13y agoThe original source is taken from a medical image DICOM viewer. In my limited experience as a medical physics student, the people working with these tools would really benefit from a comment explaining the code. They are most definitely not coders, most of them having barely done anything more than write a few matlab scripts.
- auvrw 13y agolike dict(zip( or dict(getmembers(asdf)).keys(), it's idiomatic code. it wouldn't have occurred to me the first time i had to write such a function, but now that i've taken a few moments to read the article i find it clearer than the list comprehension version (because the constant only appears once) and nicer than the numpy version in that it doesn't require an extra dependency. may save a few keystrokes some rainy day. good post.
- raphael_kimmig 13y agoYou can also do zip(arr[::3], arr[1::3], arr[2::3]) which is nearly as fast but doesn't work with iterators. If you want to use iterators you could also do zip(islice(arr, 0, None, 3), islice(arr, 1, None, 3), islice(arr, 2, None, 3)) which is a tad slower.
- s_q_b 13y agoIt wouldn't have occurred to me to do this a different way. Isn't this a very basic use of zip()?
- deleted 13y ago[deleted]
- Jach 13y agoI'm glad Clojure has top-level support for this operation... it's quite flexible too, and the presence of partition-all makes it explicit what you should expect if the sequence doesn't evenly partition. user=> (partition 3 [1 2 3 4 5 6]) ((1 2 3) (4 5 6)) user=> (partition 3 [1 2 3 4 5 6 7]) ((1 2 3) (4 5 6)) user=> (partition-all 3 [1 2 3 4 5 6 7]) ((1 2 3) (4 5 6) (7)) user=> (partition 3 3 (repeat 0) [1 2 3 4 5 6 7]) ((1 2 3) (4 5 6) (7 0 0))
- iamgopal 13y agocan you measure the speed and post it here ?
- anaphor 13y agoHaskell also has it in the Data.List.Split library, which is handy, although it's easy to define yourself. λ> chunksOf 3 [1..12] [[1,2,3],[4,5,6],[7,8,9],[10,11,12]]
- phpnode 13y agoFAO: iamgopal - your account has been dead for over a year for no discernible reason, only people with showdead on can see your posts.
- chilldream 13y agoThe part that annoys me is that the docstring mentions "dictionaries" when I see no `dict`s.
- StavrosK 13y agoYeah, looks like a remnant that was never updated.
- d0mine 13y agoFrom Itertools Recipes [6]: def grouper(iterable, n, fillvalue=None): "Collect data into fixed-length chunks or blocks" # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx" args = [iter(iterable)] * n return zip_longest(*args, fillvalue=fillvalue) - What is the most “pythonic” way to iterate over a list in chunks? [1] - Idiomatic way to take groups of n items from a list in Python? [2] - Python “Every Other Element” Idiom [3] - Iterate an iterator by chunks (of n) in Python? [4] - How do you split a list into evenly sized chunks in Python? [5] [1]: http://stackoverflow.com/questions/434287/what-is-the-most-pythonic-way-to-iterate-over-a-list-in-chunks http://stackoverflow.com/questions/434287/what-is-the-most-p... [2]: http://stackoverflow.com/questions/2461484/idiomatic-way-to-take-groups-of-n-items-from-a-list-in-python http://stackoverflow.com/questions/2461484/idiomatic-way-to-... [3]: http://stackoverflow.com/questions/2631189/python-every-other-element-idiom http://stackoverflow.com/questions/2631189/python-every-othe... [4]: http://stackoverflow.com/questions/8991506/iterate-an-iterator-by-chunks-of-n-in-python http://stackoverflow.com/questions/8991506/iterate-an-iterat... [5]: http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python http://stackoverflow.com/questions/312443/how-do-you-split-a... [6]: http://docs.python.org/3/library/itertools.html#itertools-recipes http://docs.python.org/3/library/itertools.html#itertools-re...
- iooi 13y agoFigured I would add my own [7]. I know the example prompt might seem a bit specific, but it can come in really handy sometimes (for example, iterating through get parameters). Always a fan of idiomatic and one-liner python. I should mention that I ended up using the fourth version (seemingly the slowest) but it is actually the fastest depending on your input -- as the length of the elements gets larger, the fourth method tends to vastly outperform the others. [7] http://stackoverflow.com/questions/16685545/elegantly-iterating-over-string-replacement http://stackoverflow.com/questions/16685545/elegantly-iterat...
- davidp 13y agoI made a few more interesting (to me) measurements. As always, you have to measure your performance with your actual input data to see what's "best". Test 1: Boring, small array of integers In [28]: arr = range(0, 300) In [29]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)] 10000 loops, best of 3: 27.2 us per loop In [30]: %timeit numpy.reshape(arr, (-1, 3)) 10000 loops, best of 3: 45.2 us per loop In [31]: %timeit zip(*([iter(arr)]*3)) 100000 loops, best of 3: 6.25 us per loop This roughly matches the article's timing ratios, so far so good. Test 2: Use numpy's random number generation to get a small array of floats In [32]: arr = numpy.random.ranf(300) In [33]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)] 10000 loops, best of 3: 54 us per loop In [34]: %timeit numpy.reshape(arr, (-1, 3)) 1000000 loops, best of 3: 1.06 us per loop In [35]: %timeit zip(*([iter(arr)]*3)) 10000 loops, best of 3: 39.7 us per loop numpy is two orders of magnitude faster here; it's evidently using a highly optimized internal codepath for random sequence generation, which I'd guess is a common thing to do in numeric analysis. I assume it's using a generator, so there's no actual array being created, blowing up the CPU cache lines etc. Test 3: Verify that analysis by interfering with numpy In [36]: arr = [x for x in numpy.random.ranf(300)] In [37]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)] 10000 loops, best of 3: 26.2 us per loop In [38]: %timeit numpy.reshape(arr, (-1, 3)) 10000 loops, best of 3: 48.5 us per loop In [39]: %timeit zip(*([iter(arr)]*3)) 100000 loops, best of 3: 6.55 us per loop Yep. Test 4: Larger data set, no interference In [40]: arr = numpy.random.ranf(3000000) In [41]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)] 1 loops, best of 3: 624 ms per loop In [42]: %timeit numpy.reshape(arr, (-1, 3)) 1000000 loops, best of 3: 1.06 us per loop In [43]: %timeit zip(*([iter(arr)]*3)) 1 loops, best of 3: 335 ms per loop The numpy time doesn't change at all from test 2 despite the larger size, but the others suffer. Again, I suspect numpy is being intelligent here; my guess is that it doesn't actually apply the function and generate the real output, it just wraps the random generator in another one. Test 5: Larger data set, interfering with numpy In [44]: arr = [x for x in numpy.random.ranf(3000000)] In [45]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)] 1 loops, best of 3: 321 ms per loop In [46]: %timeit numpy.reshape(arr, (-1, 3)) 1 loops, best of 3: 354 ms per loop In [47]: %timeit zip(*([iter(arr)]*3)) 10 loops, best of 3: 83.6 ms per loop There we go; we're back to roughly the original timing ratios. So, surprise! You always have to measure. Measure, measure measure. My bias is to write code first for legibility and modifiability, and then optimize hot spots if needed (and add comments, please, when you do so). Without doing deeper analysis I'd say one moral of the Python story is, this shows the potential power of generators. But in real-world data sets this isn't always ideal -- is it faster to load up the whole data set in memory and blast through it, or load it from disk on demand with a generator? In really high performance scenarios, is it faster to preprocess the data to fit into the CPU's cache lines? You can't tell without measuring, and you have to measure in the environment you're deploying to, since the answer may be different on a machine with 1GB RAM vs. one with 128GB RAM, or 32KB L1 cache vs. 8KB.
- joshcorbin 13y agoEven more interesting to me: In [3]: ar = [1, 2, 3, 2, 4, 6, 3, 5 ,7, 3, 5, 8] In [4]: %timeit zip([iter(ar)]3) 100000 loops, best of 3: 2.02 us per loop In [5]: %timeit zip(ar[0::3], ar[1::3], ar[2::3]) 1000000 loops, best of 3: 1.37 us per loop In [6]: %timeit zip((iter(ar),)3) 1000000 loops, best of 3: 1.34 us per loop From which I conclude: - zipping slices is even more efficient, and arguably easier to grok - but you get about the same runtime by multiplying a singleton tuple rather than a list However if you want to generalize the chunk size, multiplication seems to win out over slicing (with tuples still being more efficient than lists): In [7]: chunk1 = lambda n, it: zip([iter(it)]n) In [8]: chunk2 = lambda n, it: zip((iter(it),)n) In [9]: chunk3 = lambda n, seq: zip(*(seq[i::n] for i in xrange(n))) In [10]: %timeit chunk1(3, ar) 100000 loops, best of 3: 2.32 us per loop In [11]: %timeit chunk2(3, ar) 1000000 loops, best of 3: 1.83 us per loop In [12]: %timeit chunk3(3, ar) 100000 loops, best of 3: 3.55 us per loop
- Luyt 13y agoI wouldn't call it insane, neither brilliant. I use this function to split a sequence into pairs (or triplets, or fourths, etc): def paired(t, size=2, default=None): it = iter(t) return itertools.izip_longest(*[it]*size, fillvalue=default) I use it in a formatter which outputs alphabetized data in columns, where the order should run down the columns instead rowwise.
- rzimmerman 13y agoIf it's actually faster, the speed actually matters, and you wrap it with a well commented explanation and descriptive name, I'd consider it reasonable. Otherwise, just write out what you're doing. It's definitely clever, but fewer lines of code is not an optimization.
- sivanmz 13y agoThe pairwise recipe function in itertools demonstrates just that (using tee instead of multiplying the iterator): http://docs.python.org/2/library/itertools.html http://docs.python.org/2/library/itertools.html def pairwise(iterable): "s -> (s0,s1), (s1,s2), (s2, s3), ..." a, b = tee(iterable) next(b, None) return izip(a, b)
- runn1ng 13y ago>"there should be one — and preferably only one — obvious way to do it" Oh. Well. OK.
- BasDirks 13y agoAn aside: the animated expansion of the code blocks make the site feel twitchy
- StavrosK 13y agoHow do you mean?
- frugalmail 13y agoMy vote: Code is unreadable crap written to be discarded later. Spend some time typing it out. You're writing for the x # of people (including the author) that will have to read it countless times in the future. I would be better if they didn't have to think about the "cleverness" when they come across it. Unless of course, you're participating in an obfuscated code contest.
- rouzh 13y agoMy question is, at what point does it become incumbent on the language to provide a more meaningful idiom for this kind of expression? I think the point at which constructs like this leak into your official documentation is a pretty good line to start thinking about it. :)
- wcarss 13y agoMaybe I'm misreading it, but to me the far more heinous crime than using difficult code (setting aside that it's in the docs) is that the docstring is a total lie. """Parses an array of xyz points and returns a array of point dictionaries.""" 'Only, it doesn’t really. It takes an iterable of points...and returns an iterable of 3-tuples of groupped points' The wrongness of it would cause me to double-take, because even if I were familiar with this usage, it isn't what the comment suggests is happening. A docstring more like: """Parses an iterable of values [x,y,z,x,y,z...] and returns an iterable of 3-tuples: [(x,y,z),(x,y,z)...]""" Would be a lot more clear by simple virtue of truth, even if it didn't explain the code step by step.