4 ms·

What's so useful about iterators and generators. The article says how to use them but not why. If you already know how to make list comprehensions and use "for

What's so useful about iterators and generators. The article says how to use them but not why. If you already know how to make list comprehensions and use "for elt in coll", do they let you do anything new?

xapata 3y ago
Fundamentally? No, it's all just machine language in the end. Practically? Yes, generators are often more memory-efficient, and thereby often more compute-efficient.
- Jeff_Brown 3y ago
  Is it only about efficiency, and not expressivity? If so, can the compiler figure out where to transform my equivalent structures into iterators and generators?
  - xapata 3y ago
    The expressivity gain depends on what you're left with without the feature. It was hard for me to imagine where the previous comment draws the line around iterators and generators. An optimizing compiler would need to know what actions are pure in order to rewrite code to be lazy. Python allows so much dynamism, I can only see that happening with a very sophisticated tracing JIT.
crabbone 3y ago
You could probably find more uses, but here are some: Generators allow one to easily transform code from non-buffered to buffered. Consider this example: for y in x: do(z) Now, x may be a collection that was eagerly evaluated in previous steps, let's say a list, but then you've discovered that this list is too big to fit in memory, and you want to generate it in manageable fixed-size chunks, so you replace the code that created x to make x a generator. You don't have to touch the code above -- it will work the same way because generators have the same interface as collections. Another use: generators are used to implement async / await. I personally find this idea ridiculously stupid, but a lot of people (and especially those who don't understand what it does) like it a lot. Yielding mechanism, which is a feature of generators, is the one that's used to communicate / switch between co-routines (tasks) of asyncio. Another aspect, besides bufferization is that you might want to delegate control over how much looping you want to do to a separate chunk of code. I.e. you may want to separate the generation of elements (hence, generator) from eg. filtering them, or transforming them in some way, or reducing them etc. If you didn't have this ability, you'd have to generate the entire collection upfront, and if your computation takes multiple steps that you'd like to separate into different code chunks, you'd have to also generate intermediate collections, even though, potentially, you don't need some of the elements in those collections. Consider, for example: def powers(start, end, power): return (x ** power for x in range(start, end)) def flt(a, b, c): return a + b == c def disprove_flt(upto, upto_power): for power in range(upto_power): for a, b, c in zip(powers(1, upto - 2, power), powers(2, upto - 1, power), powers(3, upto, power)): if flt(a, b, c): return a, b, c return None, None, None Which is, of course not a correct way to search for the counterexample to Fermat's last theorem, but I tried to find a popular enough subject so that the example was easier to follow. An exercise to the reader: rewrite the code above in such a way as to eliminate "upto" and "upto_power", i.e. to search until a counterexample is found (or indefinitely).
jejones3141 3y ago
It's the difference between evaluating all the elements of a list, consing them, and returning it all at once and evaluating the elements one at a time (i.e. lazy evaluation). Saves memory and time and it gives you a chance to quit early.

xapata 3y ago

Fundamentally? No, it's all just machine language in the end. Practically? Yes, generators are often more memory-efficient, and thereby often more compute-efficient.

Jeff_Brown 3y ago
Is it only about efficiency, and not expressivity? If so, can the compiler figure out where to transform my equivalent structures into iterators and generators?
- xapata 3y ago
  The expressivity gain depends on what you're left with without the feature. It was hard for me to imagine where the previous comment draws the line around iterators and generators. An optimizing compiler would need to know what actions are pure in order to rewrite code to be lazy. Python allows so much dynamism, I can only see that happening with a very sophisticated tracing JIT.

crabbone 3y ago

You could probably find more uses, but here are some: Generators allow one to easily transform code from non-buffered to buffered. Consider this example: for y in x: do(z) Now, x may be a collection that was eagerly evaluated in previous steps, let's say a list, but then you've discovered that this list is too big to fit in memory, and you want to generate it in manageable fixed-size chunks, so you replace the code that created x to make x a generator. You don't have to touch the code above -- it will work the same way because generators have the same interface as collections. Another use: generators are used to implement async / await. I personally find this idea ridiculously stupid, but a lot of people (and especially those who don't understand what it does) like it a lot. Yielding mechanism, which is a feature of generators, is the one that's used to communicate / switch between co-routines (tasks) of asyncio. Another aspect, besides bufferization is that you might want to delegate control over how much looping you want to do to a separate chunk of code. I.e. you may want to separate the generation of elements (hence, generator) from eg. filtering them, or transforming them in some way, or reducing them etc. If you didn't have this ability, you'd have to generate the entire collection upfront, and if your computation takes multiple steps that you'd like to separate into different code chunks, you'd have to also generate intermediate collections, even though, potentially, you don't need some of the elements in those collections. Consider, for example: def powers(start, end, power): return (x ** power for x in range(start, end)) def flt(a, b, c): return a + b == c def disprove_flt(upto, upto_power): for power in range(upto_power): for a, b, c in zip(powers(1, upto - 2, power), powers(2, upto - 1, power), powers(3, upto, power)): if flt(a, b, c): return a, b, c return None, None, None Which is, of course not a correct way to search for the counterexample to Fermat's last theorem, but I tried to find a popular enough subject so that the example was easier to follow. An exercise to the reader: rewrite the code above in such a way as to eliminate "upto" and "upto_power", i.e. to search until a counterexample is found (or indefinitely).

jejones3141 3y ago

It's the difference between evaluating all the elements of a list, consing them, and returning it all at once and evaluating the elements one at a time (i.e. lazy evaluation). Saves memory and time and it gives you a chance to quit early.