3 ms·

The author conflates generators with generator expressions[1]. He's correct in that generator expressions lazily evaluate and lists don't, but actual generators

The author conflates generators with generator expressions[1]. He's correct in that generator expressions lazily evaluate and lists don't, but actual generators are a much more powerful construct - they're functions which use the `yield` keyword. They're most frequently used as iterators, but they can also communicate with the outside state via receiving values (`x = yield y`), being explicitly closed with `.close()`, or having an exception raised at the yield point with `.throw()`. They're so powerful in fact, that asyncio is implemented under the hood with a lot of generator cleverness. You rarely see it, but the `__await__` dunder method which powers `await` is just a synchronous function which returns a generator. Generator expressions are useful for when there's memory constraints that preclude using a list comprehension, but list comprehensions are actually faster because of cPython's internal optimizations. [0] https://peps.python.org/pep-0255/ https://peps.python.org/pep-0255/ [1] https://peps.python.org/pep-0289/ https://peps.python.org/pep-0289/

enragedcacti 4y ago
> but they can also communicate with the outside state via receiving values (`x = yield y`) I'm dumbfounded that I have written python for as long as I have without ever encountering this, TIL! https://peps.python.org/pep-0342/ https://peps.python.org/pep-0342/
- photochemsyn 4y ago
  Just found an interesting example explaining how that works, I wasn't aware of it either (this site also has nice 'yield from' explainer): https://lerner.co.il/2020/05/08/making-sense-of-generators-coroutines-and-yield-from-in-python/ https://lerner.co.il/2020/05/08/making-sense-of-generators-c... > "The above code looks a bit weird, in that “yield’ is on the right side of an assignment statement. This means that “yield” must be providing a value to the generator. Where is it getting that value from? Answer: From the “send” method, which can be invoked in place of the “next” function. The “send” method works just like “next”, except that you can pass any Python data structure you want into the generator. And whatever you send to the generator is then assigned to “x”."
  - memco 4y ago
    Knew about it, but didn't quite understand how to use send until I read through the link you posted. Thanks for that! Regarding that article specifically I think it could've done better service to 'yield from' had it mentioned recursive generators. One example would be flattening nested lists: have a flatten function that checks if its argument is iterable: if not just yield the thing else yield from flatten for every item in the iterable.
  - deleted 4y ago
    [deleted]
- knlb2022 4y ago
  I used this in AoC for the first time a couple of years ago, as a consequence of diving deeply into asyncio. https://explog.in/static/aoc2019/AoC23.html https://explog.in/static/aoc2019/AoC23.html
  - memco 4y ago
    I'm glad to see someone with the same idea was able to take it to completion. Life got in the way for me and I never got to implement that part. If I ever get back around to it I will consult your solution if I get stuck.
- tomn 4y ago
  Back in the days before async/await, twisted (an asynchronous I/O framework) used these to emulate async/await -- you could yield a differed value (like a promise/future), and it would pass it back when the value was available, saving some effort making your state explicit and writing many callbacks: https://twisted.org/documents/21.2.0/api/twisted.internet.defer.html#inlineCallbacks https://twisted.org/documents/21.2.0/api/twisted.internet.de...
nijave 4y ago
Besides memory, they're also great for an expensive operation. You can do pagination with network calls, disk I/O, etc
henrydark 4y ago
Generators and generator expressions are useful every time your api involves an unbounded sequence. I once wrote a library of signal processing components based on the premise that the raw input signal is such a sequence, and its concrete type was Iterable[np.ndarray] (iterable of chunks of audio signal). Since the signal is unbounded - we just continue sampling from, say, a microphone forever - the types can't involve finite sequences like lists. A general iterable is works here, and then components that transform the sequence are themselves implemented as generators, or if they are very simple, as generator expressions. Consider the following function for amplifying a signal: def amplify(signal: Iterable[np.ndarray], a: float): return (a*chunk for chunk in signal) You can flip the whole infinite streams interface and talk about "pushing" chunks around, and it's an esthetic choice.
charlieflowers 4y ago
To be fair, I don’t see a conflation here. He chose not to elaborate on generator functions, but that appears to be due to a defensible conscious choice about article scope. Imo it’s a well written, technically accurate article.

enragedcacti 4y ago

> but they can also communicate with the outside state via receiving values (`x = yield y`) I'm dumbfounded that I have written python for as long as I have without ever encountering this, TIL! https://peps.python.org/pep-0342/ https://peps.python.org/pep-0342/

photochemsyn 4y ago
Just found an interesting example explaining how that works, I wasn't aware of it either (this site also has nice 'yield from' explainer): https://lerner.co.il/2020/05/08/making-sense-of-generators-coroutines-and-yield-from-in-python/ https://lerner.co.il/2020/05/08/making-sense-of-generators-c... > "The above code looks a bit weird, in that “yield’ is on the right side of an assignment statement. This means that “yield” must be providing a value to the generator. Where is it getting that value from? Answer: From the “send” method, which can be invoked in place of the “next” function. The “send” method works just like “next”, except that you can pass any Python data structure you want into the generator. And whatever you send to the generator is then assigned to “x”."
- memco 4y ago
  Knew about it, but didn't quite understand how to use send until I read through the link you posted. Thanks for that! Regarding that article specifically I think it could've done better service to 'yield from' had it mentioned recursive generators. One example would be flattening nested lists: have a flatten function that checks if its argument is iterable: if not just yield the thing else yield from flatten for every item in the iterable.
- deleted 4y ago
  [deleted]
knlb2022 4y ago
I used this in AoC for the first time a couple of years ago, as a consequence of diving deeply into asyncio. https://explog.in/static/aoc2019/AoC23.html https://explog.in/static/aoc2019/AoC23.html
- memco 4y ago
  I'm glad to see someone with the same idea was able to take it to completion. Life got in the way for me and I never got to implement that part. If I ever get back around to it I will consult your solution if I get stuck.
tomn 4y ago
Back in the days before async/await, twisted (an asynchronous I/O framework) used these to emulate async/await -- you could yield a differed value (like a promise/future), and it would pass it back when the value was available, saving some effort making your state explicit and writing many callbacks: https://twisted.org/documents/21.2.0/api/twisted.internet.defer.html#inlineCallbacks https://twisted.org/documents/21.2.0/api/twisted.internet.de...

nijave 4y ago

Besides memory, they're also great for an expensive operation. You can do pagination with network calls, disk I/O, etc

henrydark 4y ago

Generators and generator expressions are useful every time your api involves an unbounded sequence. I once wrote a library of signal processing components based on the premise that the raw input signal is such a sequence, and its concrete type was Iterable[np.ndarray] (iterable of chunks of audio signal). Since the signal is unbounded - we just continue sampling from, say, a microphone forever - the types can't involve finite sequences like lists. A general iterable is works here, and then components that transform the sequence are themselves implemented as generators, or if they are very simple, as generator expressions. Consider the following function for amplifying a signal: def amplify(signal: Iterable[np.ndarray], a: float): return (a*chunk for chunk in signal) You can flip the whole infinite streams interface and talk about "pushing" chunks around, and it's an esthetic choice.

charlieflowers 4y ago

To be fair, I don’t see a conflation here. He chose not to elaborate on generator functions, but that appears to be due to a defensible conscious choice about article scope. Imo it’s a well written, technically accurate article.