5 ms·

Isn’t there something called LazyEnumerator that deals with this? In terms of chained .map at least

djur 9y ago
Lazy enumerators don't make each individual step of the iteration any more efficient. They just let you stop the iteration partway through without traversing the entire original enumerable. Given this contrived example: a = (1..1000).lazy.select(&:even?).map{|i| i*2 }.map{|i| i + 1 }.map(&:to_s).map(&:reverse) There is no benefit from lazy if you do `a.join(",")`, since that iterates over the entire sequence. But if you do `a.take(10)` or `a.detect{|s| s == "14" }`, the chain will only be executed for each item in turn until 10 elements are produced or an element is the string "14", respectively.
- bbatha 9y ago
  There is absolutely a benefit to lazy if you're doing more than one combinator even if you ultimately need to iterate over the whole list such as the join case. The lazy version basically turns into a = "" (1..1000).each { |i| next if i.odd? i *= 2 i += 1 a << i.to_s.reverse + ", " # ya ya, trailing comma } Where as the non lazy version turns into: odds = [] (1..1000).each { |i| odds << i if i.even? } doubled = [] odds.each { |o| doubled << o * 2 } incremented = [] doubled.each { |d| incremented << d + 1 } strs = [] incremented.each { |i| i.to_s } a = "" strs.each{ |s| a << s.reversed + ", " } Which means that you have you went from N iterations to 5*N iterations, with 5 intermediate arrays in this case.
  - djur 9y ago
    That's how it could work theoretically, but in practice lazy iterators are much less efficient due to their implementation. I benchmarked it here: https://gist.github.com/mboeh/bb480d93c71046e23816f2c24c23e3b4 https://gist.github.com/mboeh/bb480d93c71046e23816f2c24c23e3... When applied to the same amount of work, lazy iterators consume more memory and take more CPU time than the equivalent non-lazy chained iteration. There's a place for lazy iterators, but you should pretty much always profile first. If you have performance problems due to a long chain of iterators, you're always going to get better results by merging some of those iterators into a single block than by making the whole chain lazy. The best use case for lazy iterators is if you're trying to avoid having the original collection in memory (if it's being streamed from a file or the like). And at that point, if you're trying to optimize for performance, you should avoid chained iterations, period. The good news is that in the vast majority of cases this just doesn't matter. Chaining iterators works fine until you start running into performance problems.
    - mullsork 9y ago
      Thanks a lot for the insight!
- cutler 9y ago
  Ruby's composabilty really puts Python to shame. I can't imagine how Ruby lost out to Python.
  - blunte 9y ago
    While I much prefer writing Ruby to Python, Python is just dead simple to get real work done with. It's not elegant, but it's still so much less nasty than C++ or Java. Ruby's #1 problem, for me at least, is that your small project someday runs into a performance wall. I don't know the latest benchmarks, but last I recall Python was about 8x faster than Ruby when both are being interpreted. Yes, there's JRuby, but that's not something you can drop into a new system and do useful things with immediately (without more setup). And with Clojure, you get all the elegance and lovely collection manipulation tools you can possibly want, much faster performance, and a huge stable pile of Java libraries (compared to Ruby). So with Python and Clojure as one's main tools, life is quite nice.
    - yxhuvud 9y ago
      Say what? Ruby has been faster than Python for interpreted code for years now. Granted, Python is faster for tasks like those Numpy solves, but in arbitrary execution performance, python is just slower. 8x has NEVER been true. That must have been you doing some really bad stuff in one of them but not both.
      - jashmatthews 9y ago
        I think there was a short period about 11 years ago before the release of Ruby 1.9 where the Python VM was basically re-written to be as fast as YARV, but YARV wasn't released yet.
      - blunte 9y ago
        I stand corrected. I used Ruby for 6 years, up to 2015. I wasn't using Python during those years, but I thought I had heard or read about the 8x speed difference in favor of Python. For sure, Ruby was slow for what we were doing... processing millions of rows of data was so slow that it caused me to decide to try Go (and the same Go program was hundreds of times faster). But I didn't try Python then.

djur 9y ago

Lazy enumerators don't make each individual step of the iteration any more efficient. They just let you stop the iteration partway through without traversing the entire original enumerable. Given this contrived example: a = (1..1000).lazy.select(&:even?).map{|i| i*2 }.map{|i| i + 1 }.map(&:to_s).map(&:reverse) There is no benefit from lazy if you do `a.join(",")`, since that iterates over the entire sequence. But if you do `a.take(10)` or `a.detect{|s| s == "14" }`, the chain will only be executed for each item in turn until 10 elements are produced or an element is the string "14", respectively.

bbatha 9y ago
There is absolutely a benefit to lazy if you're doing more than one combinator even if you ultimately need to iterate over the whole list such as the join case. The lazy version basically turns into a = "" (1..1000).each { |i| next if i.odd? i *= 2 i += 1 a << i.to_s.reverse + ", " # ya ya, trailing comma } Where as the non lazy version turns into: odds = [] (1..1000).each { |i| odds << i if i.even? } doubled = [] odds.each { |o| doubled << o * 2 } incremented = [] doubled.each { |d| incremented << d + 1 } strs = [] incremented.each { |i| i.to_s } a = "" strs.each{ |s| a << s.reversed + ", " } Which means that you have you went from N iterations to 5*N iterations, with 5 intermediate arrays in this case.
- djur 9y ago
  That's how it could work theoretically, but in practice lazy iterators are much less efficient due to their implementation. I benchmarked it here: https://gist.github.com/mboeh/bb480d93c71046e23816f2c24c23e3b4 https://gist.github.com/mboeh/bb480d93c71046e23816f2c24c23e3... When applied to the same amount of work, lazy iterators consume more memory and take more CPU time than the equivalent non-lazy chained iteration. There's a place for lazy iterators, but you should pretty much always profile first. If you have performance problems due to a long chain of iterators, you're always going to get better results by merging some of those iterators into a single block than by making the whole chain lazy. The best use case for lazy iterators is if you're trying to avoid having the original collection in memory (if it's being streamed from a file or the like). And at that point, if you're trying to optimize for performance, you should avoid chained iterations, period. The good news is that in the vast majority of cases this just doesn't matter. Chaining iterators works fine until you start running into performance problems.
  - mullsork 9y ago
    Thanks a lot for the insight!
cutler 9y ago
Ruby's composabilty really puts Python to shame. I can't imagine how Ruby lost out to Python.
- blunte 9y ago
  While I much prefer writing Ruby to Python, Python is just dead simple to get real work done with. It's not elegant, but it's still so much less nasty than C++ or Java. Ruby's #1 problem, for me at least, is that your small project someday runs into a performance wall. I don't know the latest benchmarks, but last I recall Python was about 8x faster than Ruby when both are being interpreted. Yes, there's JRuby, but that's not something you can drop into a new system and do useful things with immediately (without more setup). And with Clojure, you get all the elegance and lovely collection manipulation tools you can possibly want, much faster performance, and a huge stable pile of Java libraries (compared to Ruby). So with Python and Clojure as one's main tools, life is quite nice.
  - yxhuvud 9y ago
    Say what? Ruby has been faster than Python for interpreted code for years now. Granted, Python is faster for tasks like those Numpy solves, but in arbitrary execution performance, python is just slower. 8x has NEVER been true. That must have been you doing some really bad stuff in one of them but not both.
    - jashmatthews 9y ago
      I think there was a short period about 11 years ago before the release of Ruby 1.9 where the Python VM was basically re-written to be as fast as YARV, but YARV wasn't released yet.
    - blunte 9y ago
      I stand corrected. I used Ruby for 6 years, up to 2015. I wasn't using Python during those years, but I thought I had heard or read about the 8x speed difference in favor of Python. For sure, Ruby was slow for what we were doing... processing millions of rows of data was so slow that it caused me to decide to try Go (and the same Go program was hundreds of times faster). But I didn't try Python then.