4 ms·

Please excuse my ignorance, but maybe someone can help me convince that I'm wrong. This article illustrates a problem I have with FP in general, coming from pr

Please excuse my ignorance, but maybe someone can help me convince that I'm wrong. This article illustrates a problem I have with FP in general, coming from procedural / OOP codebases: For the generator example, how can I be sure that the initial generation of strings isn't getting executed twice? Will python analyse that I'm only ever reading from "left" or "right" within the gen_uc / gen_lc loops, thus optimizing away the the other side in the upstream generator? So I'm basically trusting the runtime to optimize for me? Do we have benchmark showing the overhead for this runtime analysis? Does it work consistently with JITed pythons like pypy? So far I'm just always too skeptical that I could program myself into a performance hole that require complete rewrites to get out of, so I stick with code where I know what will happen on the machine. Python for me is just the toy language to try these concepts before diving into actually performant ones, but not finding any significant Haskell/F#/.. benchmarks that can perform as well as good C or Fortran code, hasn't helped in reducing my skepticism. IMO the thinking of "correctness now, performance later" is a fallacy in that the choices I'm making in the design phase can lead to complete rewrites in order to get something performant - in the worst case switching between languages.

icebraining 11y ago
You can be sure it doesn't execute twice because the generation is lazy/on-demand. When you create a generator, nothing gets actually done, it just sits there waiting for you to pull elements from it, and which point it'll actually generate one element (and wait for you to request another). This is in contrast with list comprehensions, which have a similar syntax, but which generate a full list immediately after being declared.
- m_mueller 11y ago
  I know that this is true in general, but this case here seems tricky to me. Look at the following line: > yield (s[len(s)//2:], s[:len(s)//2]) I'm aware that the generator is being called lazily, but here we're actually calling each yield twice - once the left side is thrown away, once the right side downstream. Now, are you still sure that left and right is generated only once?
  - icebraining 11y ago
    Oh, right, I missed that. No, it's actually generating both every time. CPython can't eliminate one of the sides, since the calls (either to len() or to the slicing method of 's') might have side effects. In this case, I think it'd make more sense to divide gen_splitted_his into two functions (or one which accepted a parameter), for generating one or the other.
    - m_mueller 11y ago
      I have been toying around now with this second python example for a bit. Here's my conclusion: Not only is it not efficient, it doesn't even work. The generator cannot be rewound after it has been iterated over for lowercase - the uppercase version will never be executed. You essentially run into the problem described here: http://stackoverflow.com/questions/1271320/reseting-generator-object-in-python http://stackoverflow.com/questions/1271320/reseting-generato... . It only works once you actually generate twice > lc_his = gen_lc_his(gen_splitted_his(('hi for the %d:th time!' % i for i in xrange(10)))) > uc_his = gen_uc_his(gen_splitted_his(('hi for the %d:th time!' % i for i in xrange(10)))) It seems to me for this toy example, generators are simply not the right tool.
      - icebraining 11y ago
        Yes, but I think the author realizes that, which is why (s)he wrote above "this will produce an error, since running a generator function twice is not allowed, and this would not re-use the same items in memory anyway, which is kind of the main point with stream processing". The solution being proposed at the end is actually not using generators at all.
        m_mueller 11y ago
        Oh I see, I missed that this "limitation" also applies for his DPI example that I was toying with.

icebraining 11y ago

You can be sure it doesn't execute twice because the generation is lazy/on-demand. When you create a generator, nothing gets actually done, it just sits there waiting for you to pull elements from it, and which point it'll actually generate one element (and wait for you to request another). This is in contrast with list comprehensions, which have a similar syntax, but which generate a full list immediately after being declared.

m_mueller 11y ago
I know that this is true in general, but this case here seems tricky to me. Look at the following line: > yield (s[len(s)//2:], s[:len(s)//2]) I'm aware that the generator is being called lazily, but here we're actually calling each yield twice - once the left side is thrown away, once the right side downstream. Now, are you still sure that left and right is generated only once?
- icebraining 11y ago
  Oh, right, I missed that. No, it's actually generating both every time. CPython can't eliminate one of the sides, since the calls (either to len() or to the slicing method of 's') might have side effects. In this case, I think it'd make more sense to divide gen_splitted_his into two functions (or one which accepted a parameter), for generating one or the other.
  - m_mueller 11y ago
    I have been toying around now with this second python example for a bit. Here's my conclusion: Not only is it not efficient, it doesn't even work. The generator cannot be rewound after it has been iterated over for lowercase - the uppercase version will never be executed. You essentially run into the problem described here: http://stackoverflow.com/questions/1271320/reseting-generator-object-in-python http://stackoverflow.com/questions/1271320/reseting-generato... . It only works once you actually generate twice > lc_his = gen_lc_his(gen_splitted_his(('hi for the %d:th time!' % i for i in xrange(10)))) > uc_his = gen_uc_his(gen_splitted_his(('hi for the %d:th time!' % i for i in xrange(10)))) It seems to me for this toy example, generators are simply not the right tool.
    - icebraining 11y ago
      Yes, but I think the author realizes that, which is why (s)he wrote above "this will produce an error, since running a generator function twice is not allowed, and this would not re-use the same items in memory anyway, which is kind of the main point with stream processing". The solution being proposed at the end is actually not using generators at all.
      - m_mueller 11y ago
        Oh I see, I missed that this "limitation" also applies for his DPI example that I was toying with.