4 ms·

> I'm surprised comprehensions in particular haven't spread to more modern languages. I've come to dislike list comprehensions. Simple ones are ok but they do

by zmmmmm 6y ago

> I'm surprised comprehensions in particular haven't spread to more modern languages. I've come to dislike list comprehensions. Simple ones are ok but they do not handle incremental complexity well. Invariably it means code slowly becomes really unreadable as time goes by because nobody wants to rewrite the list comprehension when one more little tweak stuffed in there will do the job. I much prefer object-functional chaining style ala Scala, Groovy, Ruby etc: some_giant_list .findAll { it.foo > 20 } .groupBy { it.bar } .countBy { it.value.size() > 5 } They scale much better over time as new constructs get inserted as functional additions in the sequential pipeline.

earthboundkid 6y ago
Yes, as a dev who started in Python and has done more JS recently, I’ve really come to believe that the chaining of in-line functions is a better approach in the end. Python theoretically has the advantage of working with lists or iterators, but in practice, all you really use are lists.
- sammorrowdrums 6y ago
  To be fair many tools use iterator types that are not lists like Django querysets. I feel the nice thing in Python is generator expressions, so for lots of code you can decrease the memory footprint and speed some code up through easy lazy evaluation. I do appreciate the clarity of function chaining, and I think Rust made particularly good decisions around iteration in that way. Function chaining ideally needs good ways to break iteration, skip and things, and JS isn't very feature rich beyond the basic map, reduce, forEach. It feels very incomplete compared to a lot of languages that offer that stuff. Hence the popularity of underscore and then lodash.
  - earthboundkid 6y ago
    If you’re doing a list comprehension with a Django queryset, you’re risking having a million accidental DB requests
- BiteCode_dev 6y ago
  > all you really use are lists. The more you get experience in Python, and the less you use specifically lists. In fact, you can spot people getting confortable with the language when they start importing itertools, unless they are data scientists, of course.
aeturnum 6y ago
Obviously bad practice can ruin any good thing, but I much prefer list comprehensions for smaller operations and will refactor them as they become unwieldy. If you're in an environment where "nobody really wants to" rewrite stuff that's becoming hard to work with (for whatever reason) it probably doesn't matter which language feature dooms you.
fnord123 6y ago
This is only delightful when you are doing synchronous computation or you have async/await in the language. Until Loom is available, using this same pattern with CompletableFuture breaks in the face of try-with-resources. Also I don't think it's possible in Python using await since await is a prefix keyword so in your example .groupBy is not a method on Future so it won't work. I love ReactiveX as much as anyone but these issues cheese me off. Rust has await as a postfix operator so this pattern works flawlessly when mixing async and sync code.
BiteCode_dev 6y ago
This assumes some_giant_list is an object that has "findAll", that "findAll" returns and object that has "groupBy", which returns an object that has "countBy". Looking at that code, I don't even know what the intermediary objects are, but given the names of the operations, they can't be all flat lists. Besides, you may not put your list comprehension inline. It is often more readable to make it span on several lines, espacially if it's a complex one: banned_ip = { con.ip for con in connexions if con.ip in blacklist or con.type == "internal" }
- eganjs 6y ago
  This is exact how I write mine, I find it much more readable personally. The real challenge I've encountered is typically you can only use one expression. If I need to write a slightly more complex mapping I am forced to write a function, which I normally define just before the comprehension. Even though this works, it introduces boilerplate I'd rather not write. I will admit this doesn't happen often, but it happens enough to bother me. > This assumes some_giant_list is an object that has "findAll", that "findAll" returns and object that has "groupBy", which returns an object that has "countBy". > Looking at that code, I don't even know what the intermediary objects are, but given the names of the operations, they can't be all flat lists. In regards to this, I would say sure, but does that really matter. Most IDEs will give you enough inference to list the operations you can perform and the return types they give. If anything the "super collections" make a developers life far easier, I think Kotlin does a particularly good job of this with a vast set of extension functions. Have a scroll of this documentation https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/ https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collecti... mapIndexedNotNull is a great example
ascotan 6y ago
They're great until you find something like this in code: [x for y in z for x in zz if y in x] Then you begin to wonder how great they really are.
- mlthoughts2018 6y ago
  To me that looks super easy to read. You just go left to right. for y in z: for x in zz: if y in x: yield x It’s extremely easy to sight read. If the lines become long, just split them up and it’s even more obvious. [x for y in z for x in zz if y in x] There’s nothing tricky about multiple loops and conditions in comprehensions in most languages that support them. Just go left to right and it mirrors outer to inner loops/conditionals.
  - xushiwei 6y ago
    Agree.
- wenc 6y ago
  Also, most data scientists presumably have some advanced math background so it's akin to reading set-builder notation (albeit with listcomps the sets are sometimes nested). Your listcomp reads: {x : y ∈ z, x ∈ zz | y ∈ x} If you've read enough statistics/ML papers, this becomes second nature.
  - JNRowe 6y ago
    > If you've read enough statistics/ML papers, this becomes second nature. As someone who breathes that style of notation, there is excellent support in vim and emacs for custom display for readability. Conceal syntax in vim¹(builtin for rust/help/$few_others and external such as vim-cute-python²), and pretty-mode³ in emacs. I'm sure similar things exist in other editors too, but I don't use those ;) You don't get the exact representation in your example with the things I've mentioned, but if you're used to a more "mathy"-style they really are quite nice. YM[and taste]MV. 1. http://vimdoc.sourceforge.net/htmldoc/syntax.html#conceal http://vimdoc.sourceforge.net/htmldoc/syntax.html#conceal 2. https://github.com/ehamberg/vim-cute-python https://github.com/ehamberg/vim-cute-python (moresymbols branch for er… more symbols) 3. https://github.com/akatov/pretty-mode https://github.com/akatov/pretty-mode
mlthoughts2018 6y ago
Functional chaining (aka “fluent interfaces”) is very bad for dependency injection / mocking. In f(g(x)) you can directly access or patch f and g as top level names. In x.g().f() you have to patch methods internal to other structures, and this can lead to problems if the patching should only happen in a certain local scope. I encountered this recently with differences between pathlib.Path and os in Python. Consider def my_mkdir(pathname): pathlib.Path(pathname).mkdir() # or os.mkdir(pathname) From the pov of testing and decomposition, the second option is much nicer, because I can patch os.mkdir directly, and not patch pathlib.Path.mkdir (and also worry about controlling when instance creation happens to use the patch when I need it, but let other possible pathlib.Path objects my tests interacts with be constructed normally). mkdir is even a very simple example since pathlib.Path.mkdir is an instance method but only relies on the string data the instance has. Imagine how much harder if pathlib.Path.mkdir has complex interaction with the internal object structure or other instance methods. Obviously you _can_ solve it either way, but the fluent interface does nothing except require more code. On this balance I think list comprehensions (or just fmap, which is all comprehensions are) are much, much better than chaining. Also if you want to operate on data structures just operate on them with module functions. I think pandas really messes up on this. agg(groupby(df, cols), funcs) is way better than df.groupby(cols).agg(funcs)
- eterps 6y ago
  A very valuable insight, thank you.

by zmmmmm 6y ago

earthboundkid 6y ago

Yes, as a dev who started in Python and has done more JS recently, I’ve really come to believe that the chaining of in-line functions is a better approach in the end. Python theoretically has the advantage of working with lists or iterators, but in practice, all you really use are lists.

sammorrowdrums 6y ago
To be fair many tools use iterator types that are not lists like Django querysets. I feel the nice thing in Python is generator expressions, so for lots of code you can decrease the memory footprint and speed some code up through easy lazy evaluation. I do appreciate the clarity of function chaining, and I think Rust made particularly good decisions around iteration in that way. Function chaining ideally needs good ways to break iteration, skip and things, and JS isn't very feature rich beyond the basic map, reduce, forEach. It feels very incomplete compared to a lot of languages that offer that stuff. Hence the popularity of underscore and then lodash.
- earthboundkid 6y ago
  If you’re doing a list comprehension with a Django queryset, you’re risking having a million accidental DB requests
BiteCode_dev 6y ago
> all you really use are lists. The more you get experience in Python, and the less you use specifically lists. In fact, you can spot people getting confortable with the language when they start importing itertools, unless they are data scientists, of course.

aeturnum 6y ago

Obviously bad practice can ruin any good thing, but I much prefer list comprehensions for smaller operations and will refactor them as they become unwieldy. If you're in an environment where "nobody really wants to" rewrite stuff that's becoming hard to work with (for whatever reason) it probably doesn't matter which language feature dooms you.

fnord123 6y ago

This is only delightful when you are doing synchronous computation or you have async/await in the language. Until Loom is available, using this same pattern with CompletableFuture breaks in the face of try-with-resources. Also I don't think it's possible in Python using await since await is a prefix keyword so in your example .groupBy is not a method on Future so it won't work. I love ReactiveX as much as anyone but these issues cheese me off. Rust has await as a postfix operator so this pattern works flawlessly when mixing async and sync code.

BiteCode_dev 6y ago

This assumes some_giant_list is an object that has "findAll", that "findAll" returns and object that has "groupBy", which returns an object that has "countBy". Looking at that code, I don't even know what the intermediary objects are, but given the names of the operations, they can't be all flat lists. Besides, you may not put your list comprehension inline. It is often more readable to make it span on several lines, espacially if it's a complex one: banned_ip = { con.ip for con in connexions if con.ip in blacklist or con.type == "internal" }

eganjs 6y ago
This is exact how I write mine, I find it much more readable personally. The real challenge I've encountered is typically you can only use one expression. If I need to write a slightly more complex mapping I am forced to write a function, which I normally define just before the comprehension. Even though this works, it introduces boilerplate I'd rather not write. I will admit this doesn't happen often, but it happens enough to bother me. > This assumes some_giant_list is an object that has "findAll", that "findAll" returns and object that has "groupBy", which returns an object that has "countBy". > Looking at that code, I don't even know what the intermediary objects are, but given the names of the operations, they can't be all flat lists. In regards to this, I would say sure, but does that really matter. Most IDEs will give you enough inference to list the operations you can perform and the return types they give. If anything the "super collections" make a developers life far easier, I think Kotlin does a particularly good job of this with a vast set of extension functions. Have a scroll of this documentation https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/ https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collecti... mapIndexedNotNull is a great example

ascotan 6y ago

They're great until you find something like this in code: [x for y in z for x in zz if y in x] Then you begin to wonder how great they really are.

mlthoughts2018 6y ago
To me that looks super easy to read. You just go left to right. for y in z: for x in zz: if y in x: yield x It’s extremely easy to sight read. If the lines become long, just split them up and it’s even more obvious. [x for y in z for x in zz if y in x] There’s nothing tricky about multiple loops and conditions in comprehensions in most languages that support them. Just go left to right and it mirrors outer to inner loops/conditionals.
- xushiwei 6y ago
  Agree.
wenc 6y ago
Also, most data scientists presumably have some advanced math background so it's akin to reading set-builder notation (albeit with listcomps the sets are sometimes nested). Your listcomp reads: {x : y ∈ z, x ∈ zz | y ∈ x} If you've read enough statistics/ML papers, this becomes second nature.
- JNRowe 6y ago
  > If you've read enough statistics/ML papers, this becomes second nature. As someone who breathes that style of notation, there is excellent support in vim and emacs for custom display for readability. Conceal syntax in vim¹(builtin for rust/help/$few_others and external such as vim-cute-python²), and pretty-mode³ in emacs. I'm sure similar things exist in other editors too, but I don't use those ;) You don't get the exact representation in your example with the things I've mentioned, but if you're used to a more "mathy"-style they really are quite nice. YM[and taste]MV. 1. http://vimdoc.sourceforge.net/htmldoc/syntax.html#conceal http://vimdoc.sourceforge.net/htmldoc/syntax.html#conceal 2. https://github.com/ehamberg/vim-cute-python https://github.com/ehamberg/vim-cute-python (moresymbols branch for er… more symbols) 3. https://github.com/akatov/pretty-mode https://github.com/akatov/pretty-mode

mlthoughts2018 6y ago

Functional chaining (aka “fluent interfaces”) is very bad for dependency injection / mocking. In f(g(x)) you can directly access or patch f and g as top level names. In x.g().f() you have to patch methods internal to other structures, and this can lead to problems if the patching should only happen in a certain local scope. I encountered this recently with differences between pathlib.Path and os in Python. Consider def my_mkdir(pathname): pathlib.Path(pathname).mkdir() # or os.mkdir(pathname) From the pov of testing and decomposition, the second option is much nicer, because I can patch os.mkdir directly, and not patch pathlib.Path.mkdir (and also worry about controlling when instance creation happens to use the patch when I need it, but let other possible pathlib.Path objects my tests interacts with be constructed normally). mkdir is even a very simple example since pathlib.Path.mkdir is an instance method but only relies on the string data the instance has. Imagine how much harder if pathlib.Path.mkdir has complex interaction with the internal object structure or other instance methods. Obviously you _can_ solve it either way, but the fluent interface does nothing except require more code. On this balance I think list comprehensions (or just fmap, which is all comprehensions are) are much, much better than chaining. Also if you want to operate on data structures just operate on them with module functions. I think pandas really messes up on this. agg(groupby(df, cols), funcs) is way better than df.groupby(cols).agg(funcs)

eterps 6y ago
A very valuable insight, thank you.