12 ms·

> Chaining together map/reduce/filter and other functional programming constructs (lambdas, iterators, comprehensions) may be concise, but long/multiple chains

by feoren 2y ago

> Chaining together map/reduce/filter and other functional programming constructs (lambdas, iterators, comprehensions) may be concise, but long/multiple chains hurt readability This is not at all implied by anything else in the article. This feels like a common "I'm unfamiliar with it so it's bad" gripe that the author just sneaked in. Once you become a little familiar with it, it's usually far easier to both read and write than any of the alternatives. I challenge anyone to come up with a more readable example of this: var authorsOfLongBooks = books .filter(book => book.pageCount > 1000) .map(book => book.author) .distinct() By almost any complexity metric, including his, this code is going to beat the snot out of any other way of doing this. Please, learn just the basics of functional programming. You don't need to be able to explain what a Monad is (I barely can). But you should be familiar enough that you stop randomly badmouthing map and filter like you have some sort of anti-functional-programming Tourette's syndrome.

Spivak 2y ago
authors_of_long_books = set() for book in books: if len(book.pages) > 1000: authors_of_long_books.add(book.author) return authors_of_long_books You are told explicitly at the beginning what the type of the result will be, you see that it's a single pass over books and that we're matching based on page count. There are no intermediate results to think about and no function call overhead. When you read it out loud it's also it's natural, clear, and in the right order— "for each book if the book has more than 1000 pages add it to the set."
- syklemil 2y ago
  fwiw, once Python's introduced there's the third option on the table, comprehensions, which will also be suggested by linters to avoid lambdas: authors_of_long_books: set[Author] = {book.author for book in books if book.page_count > 1000} These are somewhat contentious as they can get overly complex, but for this case it should be small & clear enough for any Python programmer.
  - davidw 2y ago
    Without syntax highlighting, "book.author for book in books if book.page_count > 1000" requires a lot more effort to parse because white space like newlines is not being used to separate things out.
    - xen0 2y ago
      Set comprehensions are normal in mathematics and, barring very long complex ones, I find them the easiest to parse because they are so natural. They're just a tad more verbose in Python than mathematics because it uses words like 'for' and 'in' instead of symbols.
    - nicwolff 2y ago
      authors_of_long_books: set[Author] = { book.author for book in books if book.page_count > 1000 }
    - syklemil 2y ago
      You've had some answers already, but I also think this is a good argument for syntax highlighting. With tools like tree-sitter it's pretty easy these days to get high quality syntax highlighting, which allows us humans to receive more information in parallel. A lot of the information we pick up in our daily lives is carried through color, and being colorblind is generally seen as a disability (albeit often a mild one which can be undetected for decades). Syntax highlighting in print is more limited because of technological and economic constraints, which might leave just bold, italics and underlines on the table, while dropping color. On screens and especially in our editors where we see the most code, a lack of color is often a self-imposed limitation.
      - davidw 2y ago
        That's not the point though. If you need the syntax highlighting to quickly make out the structure, perhaps the visual layout is not as good as it could be.
        2y ago
- tremon 2y ago
  also it's natural, clear, and in the right order That isn't natural to anyone who is not intimately familiar with procedural programming. The language-natural phrasing would be "which of these books have more than thousand pages? Can you give me their authors?" -- which maps much closer to the parent's linq query than to your code.
  - bdangubic 2y ago
    That isn't natural to anyone who is not intimately familiar with procedural programming. This is not about "procedural programming" - this is exactly how this works mentally. For kicks I just asked me 11-year old kid to write down names of all the books behind her desk (20-ish) of them and give me names of authors of books that are 200 pages or more. She "procedurally" 1. took a book 2. flipped to last page to see page count 3. wrote the name of the author if page count was more than 20 The procedural is natural, it is clear and it is in the right order
    - tvier 2y ago
      You are describing how to execute the procedure, while the gp is describing what the result should be. Both are valuable, but they're very different. My personal take is that "how to execute" is more useful for lower level and finer grained control, which "what the results should be" is better for wrangling complex logic
    - skydhash 2y ago
      That's when you're doing the job, not what the mental representation of the solution. I strongly believe if you ask her to describe the task, she would go: 1. (Take the books)->(that have 200 pages or more)->(and mark down the name of the authors)->(only once)
      - bdangubic 2y ago
        I respectfully disagree. And I think one of the core reason SWEs struggle with functional-style of programming is that it is neither intuitive nor how general-joe-doe’s brain works.
- tvier 2y ago
  Much of both sides of this argument are opinion, but wrt this comment. > ... no function call overhead. This code has more function calls. O(n) vs 3 for the original
  - deleted 2y ago
    [deleted]
  - khaledh 2y ago
    That's not true. The lambdas used in the functional version are each called once for every item in the list.
    - stouset 2y ago
      No sane optimizer is going to emit the functional code as a gajillion function calls.
      - khaledh 2y ago
        True, but now you're relying on a specific implementation and optimization of the compiler, unless the language semantics explicitly say that lambdas will be inlined.
        stouset 2y ago
        This is true of literally anything and everything your compiler emits. In practice the functional style is much easier to optimize to a far greater degree than the imperative style.
        tvier 2y ago
        This is why you shouldn't get into arguments about performance on the internet without highly specified execution environments. I'm going to take my own advice and go back to work :)
      - tvier 2y ago
- feoren 2y ago
  > You are told explicitly at the beginning what the type of the result will be I would argue that's a downside: you have to pick the appropriate data structure beforehand here, whereas .distinct() picks the data structure for you. If, in the future, someone comes up with a better way of producing a distinct set of things, the functional code gets that for free, but this code is locked into a particular way of doing things. Also, .distinct() tells you explicitly what you want, whereas the intention of set() is not as immediately obvious. > There are no intermediate results to think about I could argue that there aren't really intermediate results in my example either, depending on how you think about it. Are there intermediate results in the SQL query "SELECT DISTINCT Author FROM Books WHERE Books.PageCount > 1000"? Because that's very similar to how I mentally model the functional chain. There are also intermediate results, or at least intermediate state, in your code: at any point in the loop, your set is in an intermediate state. It's not a big deal there either though: I'd argue you don't really think about that state either. > and no function call overhead That's entirely a language-specific thing, and volatile: new versions of a language may change how any of this stuff is implemented under the hood. It could be that "for ... in" happens to be a relatively expensive construct in some languages. You're probably right that the imperative code is slightly faster in most languages today, and if it has been shown via performance analysis that this particular code is a bottleneck, it makes sense to sacrifice readability in favor of performance. But it is a sacrifice in readability, and the current debate is over which is more readable in the first place. > a single pass over books Another detail that may or may not be true, and probably doesn't matter. The overhead of different forms of loops is just not what's determining the performance of almost any modern application. Also, my example could be a single pass if those methods were implemented in a lazy, "query builder" form instead of an immediately-evaluated form. In fact, whether this query should be immediately evaluated is not necessarily this function's decision. It's nice to be able to write code that doesn't care about that. My example works the same for a wide variety of things that "books" could be, and the strategy to get the answer can be different depending on what it is. It's possible the result of this code is exactly the SQL I mentioned earlier, rather than an in-memory set. There are lots of benefits to saying what you want, instead of specifying exactly how you want it.
  - megous 2y ago
- mrkeen 2y ago
  Awesome, giving it a quick scan, authors_of_long_books = set() Now I know that authors_of_long_books is the empty set. Do I need to bother reading the rest?
jltsiren 2y ago
Your example is a conceptually simple filter on a single list of items. But once the chain grows too long, the conditions become too complex, and there are too many lists/variables involved, it becomes impossible understand everything at once. In a procedural loop, you can assign an intermediate result to a variable. By giving it a name, you can forget the processing you have done so far and focus on the next steps.
- stouset 2y ago
  You don't ever need to "understand everything at once". You can read each stanza linearly. The for loop style is the approach where everything often needs to be understood all at once since the logic is interspersed throughout the entire body.
  - jltsiren 2y ago
    In the example above, you first have a list of books. Then you filter it down to books with >1000 pages. Then you map it to authors of books with >1000 pages. Then you collapse it to distinct authors of books with >1000 pages. Every step in the chain adds further complexity to the description of the things you have, until it exceeds the capacity of your working memory. Then you can no longer reason about it. The standard approach to complexity like that is to invent useful concepts and give them descriptive names. Then you can reason about the concepts themselves, without having to consider the steps you used to reach them.
    - lelandbatey 2y ago
      Folks who are familiar with chaining don't think about it in the way that you've presented. If you're familiar, it's more like: Filter to the books with >1000 pages Then their authors. Finally, distinguish those authors. If you're familiar, you don't mentally represent each link in the chain as the totality of everything that came before it _plus_ whatever operation you're doing now. You consider each link in the chain in isolation, as its inputs are the prior link and its outputs will be used in the next link. Giving a name to each one of those links in the chain isn't always necessary, and depending on how trivial the operations are, can really hurt readability. I think its very much a personal preference.
titzer 2y ago
SELECT DISTINCT author FROM books WHERE pageCount > 1000;
- __mharrison__ 2y ago
  Folks don't seem to have a problem when SQL does it. Only when code like Pandas does it...
  - mont_tag 2y ago
    Hi Matt! I've observed this phenomenon as well. When the SQL and Pandas examples are isomorphic except for shallow syntactic differences, the root cause of the complaint must either be: * that the judgment was emotional rather than substantive * or that the syntactic differences (dots and parens) actually matter
    - __mharrison__ 2y ago
      Folks really like their intermediate dataframes... for "debugging".
- beryilma 2y ago
  This is 5 times more readable than FP example above for the same computation. The FP example uses variable book(s) five times, where using it once was sufficient for SQL. Perhaps FP languages could have learned something from SQL...
  - xigoi 2y ago
    Now modify the SQL to support books having multiple authors. (In the FP example, you would just change map to flatMap.)
- YesBox 2y ago
  Scrolled to find the SQL. Such an elegant, powerful language. Really happy I chose SQLite for my game/project.
dsego 2y ago
That's not a long chain. It doesn't even have a reduce, try nesting a few reducers and see how you like it.
- aaronbrethorst 2y ago
  What is "long"?
  - brabel 2y ago
    Are you really confused when people don't think 3 operations constitutes "long"? I would guess anyone with half a brain would agree 3 operations is not long, maybe 5 or 6 and you will have many people agreeing, and above that most.
    - aquariusDue 2y ago
      Here's an abomination of my own design in Rust for example: for (index, node) in nodes .expect("Error: No blocks for the Body") .children() .expect("Error: blocks node has no children") .nodes() .iter() .enumerate() { let block = Block::new(node, index); self.blocks.push(block); }
      - maleldil 2y ago
        Why the for loop instead of mapping the enumerate iterator into Block::new and collect::Vec?
        aquariusDue 2y ago
        I lack a good answer other than it didn't occur to me, now I have some code to refactor. Thanks!
        maleldil 2y ago
        FWIW, I've had to write similar code when there's some complex "folding" operation. I use map/filter a lot, but fold/reduce/accumulate always seemed harder to understand than a for loop. I also prefer nested loops rather than nested iterators.
seeinglogic 2y ago
This comment seems unnecessarily mean-spirited... perhaps I just feel that way because I'm the person on the other end of it! I agree the code you have there is very readable, but it's not really an example of what that sentence you quoted is referencing... However I didn't spell out exactly what I meant, so please allow me to clarify. For me, roughly 5 calls in a chain is where things begin to become harder to read, which is the length of the example I used. For the meaning of "multiple", I intended that to mean if there are nested chains or if the type being operated on changes, that can slow down the rate of reading for me. Functional programming constructs can be very elegant, but it's possible to go overboard :)
- ninetyninenine 2y ago
  >For me, roughly 5 calls in a chain is where things begin to become harder to read, which is the length of the example I used. This isn't just about readability. Chaining or FP is structurally more sound. It is the more proper way to code from a architectural and structural pattern perspective. given an array of numbers 1. I want to add 5 to all numbers 2. I want to convert to string 3. I want to concat hello 4. I want to create a reduced comma seperated string 5. I want to capitalize all letters in the string. This is what a for loop would look like: // assume x is the array acc = "" for(var i = 0, i < x.length; x++) { value = x[i] + 5 value += 5 stringValue = str(value).concat(hello) acc += stringValue + "," } for (var i = 0, i < acc.length; i++) { acc[i] = capitalLetter(acc[i]) } FP: addFive(x) = [i + 5 for i in x] toString(x) = [str(i) for i in x] concatHello = [i + "hello" for i in x] reduceStrings(x) = reduce((i, acc) = acc + "," + i, x) capitalize(x) = ([capitalLetter(i) for i in x]).toString() You have 5 steps. With FP all 5 steps are reuseable. With Procedural it is not. Mind you that I know you're thinking about chaining. Chaining is eqivalent to inlining multiple operations together. So for example in that case x.map(...).map(...).map(...).reduce(...).map(...) //can be made into addFive(x) = x.map(...) toString(x)= x.map(...) ... By nature functional is modular so such syntax can easily be extracted into modules with each module given a name. The procedural code cannot do this. It is structurally unsound and tightly coupled. It's not about going overboard here. The FP simply needs to be formatted to be readable, but it is the MORE proper way to code to make your code modular general and decoupled.
yongjik 2y ago
Maybe it's because I'm not familiar with such style, but I don't like how the code hides operational details. That is, if `books` contains one billion books, and the final result should contain about a hundred authors, how much extra memory does this use for intermediate results?
- yxhuvud 2y ago
  This is a valid concern I also reacted a little bit on. One thing to note though is that it is often possible to tell such chains to be lazy and only collect the end result at the end without ever generating any intermediary arrays. Which require the author to actually have an idea how big the numbers are, but that is very often the case regardless of how you write your code.
- mrkeen 2y ago
  The best way to kick the tyres on this kind of question is to plug in something literally infinite. That way if you arrive at an answer you're probably doing something right with regard to space and time usage. For example, use all the prime numbers as an expression in your chain. import Data.Function import Data.Numbers.Primes main = do let result :: [Int] = primes & filter (startingWithDigit '5') & asPairs & map pairSum & drop 100000 & take 10 print result asPairs xs = zip xs (tail xs) pairSum (a, b) = a + b startingWithDigit d x = d == head (show x) > [100960734,100960764,100960792,100960800,100960812] > 3 MiB total memory in use (0 MB lost due to fragmentation)
agent327 2y ago
More readable? How about this: SELECT DISTINCT authors FROM books WHERE page_count > 1000;
- namaria 2y ago
  Yeah but how do you get that data written to a structure database so you get to do that?
matejn 2y ago
I like the SQL solutions people posted. But what about this one in Prolog? ?- setof(Author, Book^Pages^(book_author(Book, Author), book_pages(Book, Pages), Pages > 1000), Authors). Depending on the structure of the Prolog database, it could be shorter: ?- setof(Author, Pages^(book(_, Author, Pages), Pages > 1000), Authors).
hinkley 2y ago
I wouldn’t call 3 long. Which means you’ve picked the softball counterexample. If you were trying to play devil’s advocate, chose a longer legitimate one and show how a loop or other construct would make it better. Three dots is just a random Tuesday.
elliottkember 2y ago
Good example actually. You started with a books array, and changed the type to authors half-way through. To know the return type of the chain, I have to read through it and get to the end of each line. A longBooks array, and map(longBooks, ‘author’) wouldn’t be much longer, but would involve more distinct and meaningful phrases. I used to love doing chains! I used lodash all the time for things like this. It’s fun to write code this way. But now I see that it’s just a one-liner with line breaks.
MathMonkeyMan 2y ago
> I challenge anyone [...] select distinct author from book where pageCount > 1000;
- deleted 2y ago
  [deleted]
patrick451 2y ago
Sometimes I write code like this. Then I delete it and replace it with a for loop, because a loop is just easier to understand. This functional style is what I call write only code, because the only person who can understand it is the one who wrote it. Pandas loves this kind of method chaining, and it's one of the chief reasons pandas code is hard to read.
wegfawefgawefg 2y ago
in the most popular languages this way of programming is hurt by the syntax. this could be something more like distinct filter books .pageCount >1000 .author i think fp looks pretty terrible in js, rust, python, etc
throwA29B 2y ago
Chaining calls is an anti-pattern. Not only this is needless duplication of ye olde imperative statements sequence it also makes debugging, modifying ("oh I need to call some function in the middle of the chain, ugh"), and understanding harder for superficial benefit of it looking "cool". It actively hurts maintainability, please stop using it.
throwaway894345 2y ago
FWIW, I'm plenty familiar with functional programming and iterator chains, and I still think for loops often beat them--not only from a "visual noise" perspective, but because complex iterator chains are harder to read than equivalent for loops (particularly when you have to deal with errors-as-values and short circuiting or other patterns) and for simple tasks iterator chains might be marginally simpler but the absolute complexity of the task is so low that a for loop is fine. > But you should be familiar enough that you stop randomly badmouthing map and filter like you have some sort of anti-functional-programming Tourette's syndrome. I've been moderated for saying much tamer, FYI.
- usrusr 2y ago
  Fully agree on the easier to read part. But despite all "coffee is read more often than written" arguments, I see a lot of merit in the functional way. There are a hundred ways to write the loop slightly wrong, or with intended behavior slightly different from the regular. The functional variant: not so much. A small variation from the standard loop in functional style is visible. Very visible. Unintended ones simple won't happen, and the intended ones are an eyesore. An eyesore impossible to miss reading, whereas a subtle variation of the imperative loop is exactly that, subtle. Easy to miss. Readability advantage functional. In my book, keeping simple things simple and the not simple things not simple beats simplicity everywhere. This is actually what I consider a big drawback of functional style: often the not-simple parts are way too condensed, almost indistinguishable from trivialities. But in the loop scenario it's often the reverse. My happy place, when writing, would be an environment that has enough AST-level understanding to transform between both styles. (other advantages of functional style: skills and habits transfer to both async and parallel, the imperative loop: not so much)
williamcotton 2y ago
And in a real functional language like F#... let authorsOfLongBooks = books |> Seq.filter (fun book -> book.pageCount > 1000) |> Seq.map (fun book -> book.author) |> Seq.distinct ...you can set breakpoints anywhere in the pipeline!
RedNifre 2y ago
While I think your example is fine, I think the complaint was more about very long chains. Personally, I like to break them up to give intermediate results names, kinda like using variable names as comments: var longBooks = books.filter(book => book.pageCount > 1000) var authorsOfLongBooks = longBooks.map(book => book.author).distinct()
eitland 2y ago
This is easy to read, but in reality I have found things to typically be a bit less straightforward. Three things typically happens: 1. people who like these chains really like them. And I've seen multiple "one liner expressions" that was composed of several statements anded or ored together needing one or two line breaks. 2. when it breaks (and it does in the real world), debugging is a mess. At least last time I had to deal with it was no good way to put breakpoints in there. Maybe things have changed recently, but typically one had to rewrite it to classic programming and then debug it. 3. It trips up a lot of otherwise good programmers who hasn't seen it before.
porridgeraisin 2y ago
When the filter/mapper becomes slightly more involved as it basically always is in real life code, the regular imperative approach is much nicer.
Archelaos 2y ago
I think, an explicit type would make it even easier to grok: ISet<Author> authorsOfLongBooks = books .filter(book => book.pageCount > 1000) .map(book => book.author) .distinct() .toHashset() Or whatever the equivalent for ISet<Author> is in the respective language. Or IReadonlySet<Author> if the set should be immutable.
frankharrison 2y ago
One core reason chaining can be bad is robustness; another longevity/maintenance. Specifically around type-safety, that is knowing that the chained type is what you expect and communicating that expectation to the person who is reading the code without them needing to know the wider context of both the chained-API nor the function the chain resides in. In the context of this article, that means more complexity, and therefore less readability. I feel this is important because I have worked on many legacy code bases where bugs were found where chains were not behaving as expected, normally after attrition in some other part of the code base, and then you have to become a detective to work out the original intent. For readability chains are bad, because they can lie about their intent, especially if there’s various semantics that can be swapped. But, like any industry or code base, if their use is consistent, and the api mature/stable, they can be powerful and fast, if.

by feoren 2y ago

Spivak 2y ago

authors_of_long_books = set() for book in books: if len(book.pages) > 1000: authors_of_long_books.add(book.author) return authors_of_long_books You are told explicitly at the beginning what the type of the result will be, you see that it's a single pass over books and that we're matching based on page count. There are no intermediate results to think about and no function call overhead. When you read it out loud it's also it's natural, clear, and in the right order— "for each book if the book has more than 1000 pages add it to the set."

syklemil 2y ago
fwiw, once Python's introduced there's the third option on the table, comprehensions, which will also be suggested by linters to avoid lambdas: authors_of_long_books: set[Author] = {book.author for book in books if book.page_count > 1000} These are somewhat contentious as they can get overly complex, but for this case it should be small & clear enough for any Python programmer.
- davidw 2y ago
  Without syntax highlighting, "book.author for book in books if book.page_count > 1000" requires a lot more effort to parse because white space like newlines is not being used to separate things out.
  - xen0 2y ago
    Set comprehensions are normal in mathematics and, barring very long complex ones, I find them the easiest to parse because they are so natural. They're just a tad more verbose in Python than mathematics because it uses words like 'for' and 'in' instead of symbols.
  - nicwolff 2y ago
    authors_of_long_books: set[Author] = { book.author for book in books if book.page_count > 1000 }
  - syklemil 2y ago
    You've had some answers already, but I also think this is a good argument for syntax highlighting. With tools like tree-sitter it's pretty easy these days to get high quality syntax highlighting, which allows us humans to receive more information in parallel. A lot of the information we pick up in our daily lives is carried through color, and being colorblind is generally seen as a disability (albeit often a mild one which can be undetected for decades). Syntax highlighting in print is more limited because of technological and economic constraints, which might leave just bold, italics and underlines on the table, while dropping color. On screens and especially in our editors where we see the most code, a lack of color is often a self-imposed limitation.
    - davidw 2y ago
      That's not the point though. If you need the syntax highlighting to quickly make out the structure, perhaps the visual layout is not as good as it could be.
      - 2y ago
tremon 2y ago
also it's natural, clear, and in the right order That isn't natural to anyone who is not intimately familiar with procedural programming. The language-natural phrasing would be "which of these books have more than thousand pages? Can you give me their authors?" -- which maps much closer to the parent's linq query than to your code.
- bdangubic 2y ago
  That isn't natural to anyone who is not intimately familiar with procedural programming. This is not about "procedural programming" - this is exactly how this works mentally. For kicks I just asked me 11-year old kid to write down names of all the books behind her desk (20-ish) of them and give me names of authors of books that are 200 pages or more. She "procedurally" 1. took a book 2. flipped to last page to see page count 3. wrote the name of the author if page count was more than 20 The procedural is natural, it is clear and it is in the right order
  - tvier 2y ago
    You are describing how to execute the procedure, while the gp is describing what the result should be. Both are valuable, but they're very different. My personal take is that "how to execute" is more useful for lower level and finer grained control, which "what the results should be" is better for wrangling complex logic
  - skydhash 2y ago
    That's when you're doing the job, not what the mental representation of the solution. I strongly believe if you ask her to describe the task, she would go: 1. (Take the books)->(that have 200 pages or more)->(and mark down the name of the authors)->(only once)
    - bdangubic 2y ago
      I respectfully disagree. And I think one of the core reason SWEs struggle with functional-style of programming is that it is neither intuitive nor how general-joe-doe’s brain works.
tvier 2y ago
Much of both sides of this argument are opinion, but wrt this comment. > ... no function call overhead. This code has more function calls. O(n) vs 3 for the original
- deleted 2y ago
  [deleted]
- khaledh 2y ago
  That's not true. The lambdas used in the functional version are each called once for every item in the list.
  - stouset 2y ago
    No sane optimizer is going to emit the functional code as a gajillion function calls.
    - khaledh 2y ago
      True, but now you're relying on a specific implementation and optimization of the compiler, unless the language semantics explicitly say that lambdas will be inlined.
      - stouset 2y ago
        This is true of literally anything and everything your compiler emits. In practice the functional style is much easier to optimize to a far greater degree than the imperative style.
        tvier 2y ago
        This is why you shouldn't get into arguments about performance on the internet without highly specified execution environments. I'm going to take my own advice and go back to work :)
    - tvier 2y ago
feoren 2y ago
> You are told explicitly at the beginning what the type of the result will be I would argue that's a downside: you have to pick the appropriate data structure beforehand here, whereas .distinct() picks the data structure for you. If, in the future, someone comes up with a better way of producing a distinct set of things, the functional code gets that for free, but this code is locked into a particular way of doing things. Also, .distinct() tells you explicitly what you want, whereas the intention of set() is not as immediately obvious. > There are no intermediate results to think about I could argue that there aren't really intermediate results in my example either, depending on how you think about it. Are there intermediate results in the SQL query "SELECT DISTINCT Author FROM Books WHERE Books.PageCount > 1000"? Because that's very similar to how I mentally model the functional chain. There are also intermediate results, or at least intermediate state, in your code: at any point in the loop, your set is in an intermediate state. It's not a big deal there either though: I'd argue you don't really think about that state either. > and no function call overhead That's entirely a language-specific thing, and volatile: new versions of a language may change how any of this stuff is implemented under the hood. It could be that "for ... in" happens to be a relatively expensive construct in some languages. You're probably right that the imperative code is slightly faster in most languages today, and if it has been shown via performance analysis that this particular code is a bottleneck, it makes sense to sacrifice readability in favor of performance. But it is a sacrifice in readability, and the current debate is over which is more readable in the first place. > a single pass over books Another detail that may or may not be true, and probably doesn't matter. The overhead of different forms of loops is just not what's determining the performance of almost any modern application. Also, my example could be a single pass if those methods were implemented in a lazy, "query builder" form instead of an immediately-evaluated form. In fact, whether this query should be immediately evaluated is not necessarily this function's decision. It's nice to be able to write code that doesn't care about that. My example works the same for a wide variety of things that "books" could be, and the strategy to get the answer can be different depending on what it is. It's possible the result of this code is exactly the SQL I mentioned earlier, rather than an in-memory set. There are lots of benefits to saying what you want, instead of specifying exactly how you want it.
- megous 2y ago
mrkeen 2y ago
Awesome, giving it a quick scan, authors_of_long_books = set() Now I know that authors_of_long_books is the empty set. Do I need to bother reading the rest?

jltsiren 2y ago

Your example is a conceptually simple filter on a single list of items. But once the chain grows too long, the conditions become too complex, and there are too many lists/variables involved, it becomes impossible understand everything at once. In a procedural loop, you can assign an intermediate result to a variable. By giving it a name, you can forget the processing you have done so far and focus on the next steps.

stouset 2y ago
You don't ever need to "understand everything at once". You can read each stanza linearly. The for loop style is the approach where everything often needs to be understood all at once since the logic is interspersed throughout the entire body.
- jltsiren 2y ago
  In the example above, you first have a list of books. Then you filter it down to books with >1000 pages. Then you map it to authors of books with >1000 pages. Then you collapse it to distinct authors of books with >1000 pages. Every step in the chain adds further complexity to the description of the things you have, until it exceeds the capacity of your working memory. Then you can no longer reason about it. The standard approach to complexity like that is to invent useful concepts and give them descriptive names. Then you can reason about the concepts themselves, without having to consider the steps you used to reach them.
  - lelandbatey 2y ago
    Folks who are familiar with chaining don't think about it in the way that you've presented. If you're familiar, it's more like: Filter to the books with >1000 pages Then their authors. Finally, distinguish those authors. If you're familiar, you don't mentally represent each link in the chain as the totality of everything that came before it _plus_ whatever operation you're doing now. You consider each link in the chain in isolation, as its inputs are the prior link and its outputs will be used in the next link. Giving a name to each one of those links in the chain isn't always necessary, and depending on how trivial the operations are, can really hurt readability. I think its very much a personal preference.

titzer 2y ago

SELECT DISTINCT author FROM books WHERE pageCount > 1000;

__mharrison__ 2y ago
Folks don't seem to have a problem when SQL does it. Only when code like Pandas does it...
- mont_tag 2y ago
  Hi Matt! I've observed this phenomenon as well. When the SQL and Pandas examples are isomorphic except for shallow syntactic differences, the root cause of the complaint must either be: * that the judgment was emotional rather than substantive * or that the syntactic differences (dots and parens) actually matter
  - __mharrison__ 2y ago
    Folks really like their intermediate dataframes... for "debugging".
beryilma 2y ago
This is 5 times more readable than FP example above for the same computation. The FP example uses variable book(s) five times, where using it once was sufficient for SQL. Perhaps FP languages could have learned something from SQL...
- xigoi 2y ago
  Now modify the SQL to support books having multiple authors. (In the FP example, you would just change map to flatMap.)
YesBox 2y ago
Scrolled to find the SQL. Such an elegant, powerful language. Really happy I chose SQLite for my game/project.

dsego 2y ago

That's not a long chain. It doesn't even have a reduce, try nesting a few reducers and see how you like it.

aaronbrethorst 2y ago
What is "long"?
- brabel 2y ago
  Are you really confused when people don't think 3 operations constitutes "long"? I would guess anyone with half a brain would agree 3 operations is not long, maybe 5 or 6 and you will have many people agreeing, and above that most.
  - aquariusDue 2y ago
    Here's an abomination of my own design in Rust for example: for (index, node) in nodes .expect("Error: No blocks for the Body") .children() .expect("Error: blocks node has no children") .nodes() .iter() .enumerate() { let block = Block::new(node, index); self.blocks.push(block); }
    - maleldil 2y ago
      Why the for loop instead of mapping the enumerate iterator into Block::new and collect::Vec?
      - aquariusDue 2y ago
        I lack a good answer other than it didn't occur to me, now I have some code to refactor. Thanks!
        maleldil 2y ago
        FWIW, I've had to write similar code when there's some complex "folding" operation. I use map/filter a lot, but fold/reduce/accumulate always seemed harder to understand than a for loop. I also prefer nested loops rather than nested iterators.

seeinglogic 2y ago

This comment seems unnecessarily mean-spirited... perhaps I just feel that way because I'm the person on the other end of it! I agree the code you have there is very readable, but it's not really an example of what that sentence you quoted is referencing... However I didn't spell out exactly what I meant, so please allow me to clarify. For me, roughly 5 calls in a chain is where things begin to become harder to read, which is the length of the example I used. For the meaning of "multiple", I intended that to mean if there are nested chains or if the type being operated on changes, that can slow down the rate of reading for me. Functional programming constructs can be very elegant, but it's possible to go overboard :)

ninetyninenine 2y ago
>For me, roughly 5 calls in a chain is where things begin to become harder to read, which is the length of the example I used. This isn't just about readability. Chaining or FP is structurally more sound. It is the more proper way to code from a architectural and structural pattern perspective. given an array of numbers 1. I want to add 5 to all numbers 2. I want to convert to string 3. I want to concat hello 4. I want to create a reduced comma seperated string 5. I want to capitalize all letters in the string. This is what a for loop would look like: // assume x is the array acc = "" for(var i = 0, i < x.length; x++) { value = x[i] + 5 value += 5 stringValue = str(value).concat(hello) acc += stringValue + "," } for (var i = 0, i < acc.length; i++) { acc[i] = capitalLetter(acc[i]) } FP: addFive(x) = [i + 5 for i in x] toString(x) = [str(i) for i in x] concatHello = [i + "hello" for i in x] reduceStrings(x) = reduce((i, acc) = acc + "," + i, x) capitalize(x) = ([capitalLetter(i) for i in x]).toString() You have 5 steps. With FP all 5 steps are reuseable. With Procedural it is not. Mind you that I know you're thinking about chaining. Chaining is eqivalent to inlining multiple operations together. So for example in that case x.map(...).map(...).map(...).reduce(...).map(...) //can be made into addFive(x) = x.map(...) toString(x)= x.map(...) ... By nature functional is modular so such syntax can easily be extracted into modules with each module given a name. The procedural code cannot do this. It is structurally unsound and tightly coupled. It's not about going overboard here. The FP simply needs to be formatted to be readable, but it is the MORE proper way to code to make your code modular general and decoupled.

yongjik 2y ago

Maybe it's because I'm not familiar with such style, but I don't like how the code hides operational details. That is, if `books` contains one billion books, and the final result should contain about a hundred authors, how much extra memory does this use for intermediate results?

yxhuvud 2y ago
This is a valid concern I also reacted a little bit on. One thing to note though is that it is often possible to tell such chains to be lazy and only collect the end result at the end without ever generating any intermediary arrays. Which require the author to actually have an idea how big the numbers are, but that is very often the case regardless of how you write your code.
mrkeen 2y ago
The best way to kick the tyres on this kind of question is to plug in something literally infinite. That way if you arrive at an answer you're probably doing something right with regard to space and time usage. For example, use all the prime numbers as an expression in your chain. import Data.Function import Data.Numbers.Primes main = do let result :: [Int] = primes & filter (startingWithDigit '5') & asPairs & map pairSum & drop 100000 & take 10 print result asPairs xs = zip xs (tail xs) pairSum (a, b) = a + b startingWithDigit d x = d == head (show x) > [100960734,100960764,100960792,100960800,100960812] > 3 MiB total memory in use (0 MB lost due to fragmentation)

agent327 2y ago

More readable? How about this: SELECT DISTINCT authors FROM books WHERE page_count > 1000;

namaria 2y ago
Yeah but how do you get that data written to a structure database so you get to do that?

matejn 2y ago

I like the SQL solutions people posted. But what about this one in Prolog? ?- setof(Author, Book^Pages^(book_author(Book, Author), book_pages(Book, Pages), Pages > 1000), Authors). Depending on the structure of the Prolog database, it could be shorter: ?- setof(Author, Pages^(book(_, Author, Pages), Pages > 1000), Authors).

hinkley 2y ago

I wouldn’t call 3 long. Which means you’ve picked the softball counterexample. If you were trying to play devil’s advocate, chose a longer legitimate one and show how a loop or other construct would make it better. Three dots is just a random Tuesday.

elliottkember 2y ago

Good example actually. You started with a books array, and changed the type to authors half-way through. To know the return type of the chain, I have to read through it and get to the end of each line. A longBooks array, and map(longBooks, ‘author’) wouldn’t be much longer, but would involve more distinct and meaningful phrases. I used to love doing chains! I used lodash all the time for things like this. It’s fun to write code this way. But now I see that it’s just a one-liner with line breaks.

MathMonkeyMan 2y ago

> I challenge anyone [...] select distinct author from book where pageCount > 1000;

deleted 2y ago
[deleted]

patrick451 2y ago

Sometimes I write code like this. Then I delete it and replace it with a for loop, because a loop is just easier to understand. This functional style is what I call write only code, because the only person who can understand it is the one who wrote it. Pandas loves this kind of method chaining, and it's one of the chief reasons pandas code is hard to read.

wegfawefgawefg 2y ago

in the most popular languages this way of programming is hurt by the syntax. this could be something more like distinct filter books .pageCount >1000 .author i think fp looks pretty terrible in js, rust, python, etc

throwA29B 2y ago

Chaining calls is an anti-pattern. Not only this is needless duplication of ye olde imperative statements sequence it also makes debugging, modifying ("oh I need to call some function in the middle of the chain, ugh"), and understanding harder for superficial benefit of it looking "cool". It actively hurts maintainability, please stop using it.

throwaway894345 2y ago

FWIW, I'm plenty familiar with functional programming and iterator chains, and I still think for loops often beat them--not only from a "visual noise" perspective, but because complex iterator chains are harder to read than equivalent for loops (particularly when you have to deal with errors-as-values and short circuiting or other patterns) and for simple tasks iterator chains might be marginally simpler but the absolute complexity of the task is so low that a for loop is fine. > But you should be familiar enough that you stop randomly badmouthing map and filter like you have some sort of anti-functional-programming Tourette's syndrome. I've been moderated for saying much tamer, FYI.

usrusr 2y ago
Fully agree on the easier to read part. But despite all "coffee is read more often than written" arguments, I see a lot of merit in the functional way. There are a hundred ways to write the loop slightly wrong, or with intended behavior slightly different from the regular. The functional variant: not so much. A small variation from the standard loop in functional style is visible. Very visible. Unintended ones simple won't happen, and the intended ones are an eyesore. An eyesore impossible to miss reading, whereas a subtle variation of the imperative loop is exactly that, subtle. Easy to miss. Readability advantage functional. In my book, keeping simple things simple and the not simple things not simple beats simplicity everywhere. This is actually what I consider a big drawback of functional style: often the not-simple parts are way too condensed, almost indistinguishable from trivialities. But in the loop scenario it's often the reverse. My happy place, when writing, would be an environment that has enough AST-level understanding to transform between both styles. (other advantages of functional style: skills and habits transfer to both async and parallel, the imperative loop: not so much)

williamcotton 2y ago

And in a real functional language like F#... let authorsOfLongBooks = books |> Seq.filter (fun book -> book.pageCount > 1000) |> Seq.map (fun book -> book.author) |> Seq.distinct ...you can set breakpoints anywhere in the pipeline!

RedNifre 2y ago

While I think your example is fine, I think the complaint was more about very long chains. Personally, I like to break them up to give intermediate results names, kinda like using variable names as comments: var longBooks = books.filter(book => book.pageCount > 1000) var authorsOfLongBooks = longBooks.map(book => book.author).distinct()

eitland 2y ago

This is easy to read, but in reality I have found things to typically be a bit less straightforward. Three things typically happens: 1. people who like these chains really like them. And I've seen multiple "one liner expressions" that was composed of several statements anded or ored together needing one or two line breaks. 2. when it breaks (and it does in the real world), debugging is a mess. At least last time I had to deal with it was no good way to put breakpoints in there. Maybe things have changed recently, but typically one had to rewrite it to classic programming and then debug it. 3. It trips up a lot of otherwise good programmers who hasn't seen it before.

porridgeraisin 2y ago

When the filter/mapper becomes slightly more involved as it basically always is in real life code, the regular imperative approach is much nicer.

Archelaos 2y ago

I think, an explicit type would make it even easier to grok: ISet<Author> authorsOfLongBooks = books .filter(book => book.pageCount > 1000) .map(book => book.author) .distinct() .toHashset() Or whatever the equivalent for ISet<Author> is in the respective language. Or IReadonlySet<Author> if the set should be immutable.

frankharrison 2y ago

One core reason chaining can be bad is robustness; another longevity/maintenance. Specifically around type-safety, that is knowing that the chained type is what you expect and communicating that expectation to the person who is reading the code without them needing to know the wider context of both the chained-API nor the function the chain resides in. In the context of this article, that means more complexity, and therefore less readability. I feel this is important because I have worked on many legacy code bases where bugs were found where chains were not behaving as expected, normally after attrition in some other part of the code base, and then you have to become a detective to work out the original intent. For readability chains are bad, because they can lie about their intent, especially if there’s various semantics that can be swapped. But, like any industry or code base, if their use is consistent, and the api mature/stable, they can be powerful and fast, if.