3 ms·

One pitfall I often fall into with map, filter, etc. is that I chain them together, forgetting each of them takes another pass through the sequence. With an ex

One pitfall I often fall into with map, filter, etc. is that I chain them together, forgetting each of them takes another pass through the sequence. With an explicit loop, it's often easier to reduce multiple passes into a single pass.

V-2 11y ago
> each of them takes another pass through the sequence Not necessarily. In C# (LINQ) you have deferred execution / lazy evaluation, when possible of course - it's possible for filtering, not possible for a group-by, but that's just reality. Combine it with yield-return pattern, and it's very easy to avoid the problem you brought up.
dragonwriter 11y ago
> One pitfall I often fall into with map, filter, etc. is that I chain them together, forgetting each of them takes another pass through the sequence. Lazy maps, filters, etc. that do not do this are available in many languages (and the default in some.) Transducers provide a similar benefit, from a somewhat opposite direction (composing reduction functions rather than chaining lazy iterating functions). Other than familiarity for some people, I don't see a real edge to imperative iteration over functional operations here.
- jlarocco 11y ago
  I guess I should have been more clear. I'm aware that they do not add extra passes in some languages. But in Swift, C++, Common Lisp, OCaml, Python, Ruby, Perl, Java, Scheme, and many others, adding another map/filter/etc. will add another pass through the sequence unless you're going out of the way to avoid it.
  - dragonwriter 11y ago
    > But in Swift, C++, Common Lisp, OCaml, Python, Ruby, Perl, Java, Scheme, and many others, adding another map/filter/etc. will add another pass through the sequence unless you're going out of the way to avoid it. I think this is wrong in the case of Python -- map/filter/reduce are lazy, and so chaining them does not result in multiple iterations of the source sequence. In Ruby, its definitely sometimes wrong, because whether or not iteration is lazy depends on the particular Enumerable object; and, in the worst case, in Ruby, you don't have to go very far out of your way. In the event that the Enumerable you are operating on isn't already a lazy one, you just call its lazy method, and chain your operations after that.
  - oisdk 11y ago
    In Swift: (0...100) .filter { $0 % 2 == 0 } .map { $0 * 2 } Will pass through the sequence multiple times, yes. But to avoid it it's as easy as: (0...100) .lazy .filter { $0 % 2 == 0 } .map { $0 * 2 }
  - ta0967 11y ago
    how much of a performance drop have you measured in real world? compare my @judges = (); for my $sport (@sports) { if (is_watersport $sport) { push @judges, get_judges $sport; } } to my @judges = map get_judges, grep is_watersport, @sports; is it worth fretting over an added pass? how many sports are we dealing with anyway? and IME, nested loops induce errors in the form of accidental `O(N*M)`.
    - jlarocco 11y ago
      Your example is a straw man. IRL, I've seen this crop up while processing database results and large datasets. On several occasions I've shaved up to several minutes off of execution times by replacing chained map, filter, and zip calls that were iterating through the data multiple times with more complicated expressions that only performed the iteration once. I've also written a graphics applications where converting several chained map/filters into a single pass made interacting with the application noticeably less laggy. I'm actually surprised so many people have challenged this as a potential pitfall. I didn't pull it out of the air or from trivial examples. I've seen it happen.
  - sanderjd 11y ago
    Just to add to the chorus of "well, maybe not by default, but it's pretty easy to do", the streams library in Java 8 is lazy, and there's this library for C++14[0]. [0]: https://github.com/jscheiny/Streams https://github.com/jscheiny/Streams.

V-2 11y ago

> each of them takes another pass through the sequence Not necessarily. In C# (LINQ) you have deferred execution / lazy evaluation, when possible of course - it's possible for filtering, not possible for a group-by, but that's just reality. Combine it with yield-return pattern, and it's very easy to avoid the problem you brought up.

dragonwriter 11y ago

> One pitfall I often fall into with map, filter, etc. is that I chain them together, forgetting each of them takes another pass through the sequence. Lazy maps, filters, etc. that do not do this are available in many languages (and the default in some.) Transducers provide a similar benefit, from a somewhat opposite direction (composing reduction functions rather than chaining lazy iterating functions). Other than familiarity for some people, I don't see a real edge to imperative iteration over functional operations here.

jlarocco 11y ago
I guess I should have been more clear. I'm aware that they do not add extra passes in some languages. But in Swift, C++, Common Lisp, OCaml, Python, Ruby, Perl, Java, Scheme, and many others, adding another map/filter/etc. will add another pass through the sequence unless you're going out of the way to avoid it.
- dragonwriter 11y ago
  > But in Swift, C++, Common Lisp, OCaml, Python, Ruby, Perl, Java, Scheme, and many others, adding another map/filter/etc. will add another pass through the sequence unless you're going out of the way to avoid it. I think this is wrong in the case of Python -- map/filter/reduce are lazy, and so chaining them does not result in multiple iterations of the source sequence. In Ruby, its definitely sometimes wrong, because whether or not iteration is lazy depends on the particular Enumerable object; and, in the worst case, in Ruby, you don't have to go very far out of your way. In the event that the Enumerable you are operating on isn't already a lazy one, you just call its lazy method, and chain your operations after that.
- oisdk 11y ago
  In Swift: (0...100) .filter { $0 % 2 == 0 } .map { $0 * 2 } Will pass through the sequence multiple times, yes. But to avoid it it's as easy as: (0...100) .lazy .filter { $0 % 2 == 0 } .map { $0 * 2 }
- ta0967 11y ago
  how much of a performance drop have you measured in real world? compare my @judges = (); for my $sport (@sports) { if (is_watersport $sport) { push @judges, get_judges $sport; } } to my @judges = map get_judges, grep is_watersport, @sports; is it worth fretting over an added pass? how many sports are we dealing with anyway? and IME, nested loops induce errors in the form of accidental `O(N*M)`.
  - jlarocco 11y ago
    Your example is a straw man. IRL, I've seen this crop up while processing database results and large datasets. On several occasions I've shaved up to several minutes off of execution times by replacing chained map, filter, and zip calls that were iterating through the data multiple times with more complicated expressions that only performed the iteration once. I've also written a graphics applications where converting several chained map/filters into a single pass made interacting with the application noticeably less laggy. I'm actually surprised so many people have challenged this as a potential pitfall. I didn't pull it out of the air or from trivial examples. I've seen it happen.
- sanderjd 11y ago
  Just to add to the chorus of "well, maybe not by default, but it's pretty easy to do", the streams library in Java 8 is lazy, and there's this library for C++14[0]. [0]: https://github.com/jscheiny/Streams https://github.com/jscheiny/Streams.