5 ms·

Python Generators in Depth

171 points by webexcess 14y ago

11001 14y ago
Thank you, great write up! I'd rather see more posts like this on the front page than reviews of the latest tablets.
haberman 14y ago
Another very nice and mind-bending presentation on coroutines, that eventually takes things so far that he creates a mini "OS" with coroutines as tasks, complete with system calls and a scheduler: http://dabeaz.com/coroutines/ http://dabeaz.com/coroutines/
- webexcess 14y ago
  Thank you, I hadn't seen that. I've added that link to the bottom of my article.
  - beagle3 14y ago
    You use "assert gen.next() is None", but asserts might be dropped by optimizations, see e.g. http://docs.python.org/release/2.5.2/ref/assert.html http://docs.python.org/release/2.5.2/ref/assert.html - and your code will break. assert should only be used for debugging, and only for things that should never happen. side effects on assert is a no-no. (Also in C)
- manaskarekar 14y ago
  The talk by the same David Beazley on Co-routines: http://blip.tv/pycon-us-videos-2009-2010-2011/a-curious-course-on-coroutines-and-concurrency-part-001-2006228 http://blip.tv/pycon-us-videos-2009-2010-2011/a-curious-cour... The talk from David Mertz on Co-routines: https://www.youtube.com/watch?v=b7R3-_ViNxk https://www.youtube.com/watch?v=b7R3-_ViNxk
  - mapleoin 14y ago
    Wow that David Beazley presentation is annoying. He spends the first 10 minutes in the introduction, complaining about how he won't have enough time to go through his presentation.
    - d23 14y ago
      I didn't watch the video, but going through the slides I was absolutely fascinated. I've been craving something new and advanced like this out of python for a while.
wookietrader 14y ago
If they could only be pickled...
- webexcess 14y ago
  That's a feature! Seriously though, not being able to define a serialization method can be a limitation. However, it is always possible to turn a generator into an iterator class with whatever methods you like, but the code may end up looking totally different.
  - wookietrader 14y ago
    To me it only looks like a limitation. So yeah, you can write a custom class. But you will need to add a custom "save state" and "load state" in your class' __iter__ method. So all the benefits go away.
pekk 14y ago
These are great. If only one could write async network code with these that wasn't vastly more complicated than the equivalent gevent code.
- masklinn 14y ago
  Tulip coroutines[0] look rather good. As with C#'s async/await you have to propagate the "asyncness" to the event loop (using `yield from` with a future or another coroutine), which is not necessary in gevent, but that should be the extent of it. [0] http://www.python.org/dev/peps/pep-3156/#coroutines-and-the-scheduler http://www.python.org/dev/peps/pep-3156/#coroutines-and-the-...
  - pekk 14y ago
    The implementation described in that PEP is massive. Seems pretty hard to explain...
Xcelerate 14y ago
Somewhat off-topic, but how are generators different than returning an object with a closed over environment? For instance, his first example in CoffeeScript would be: runningAvg = -> count = 0 total = 0 send: (value) -> total += value count += 1 total/count r = runningAvg() console.log r.send 10 console.log r.send 5 console.log r.send 0 console.log r.send 0
- deleted 14y ago
  [deleted]
- csense 14y ago
  Generators can restore you to any point in the control flow, whereas your closure-based solution (or an object-oriented solution where variables are class fields) always has to start from the top of the send() function. For simple examples, closures can present an easy alternative implementation. But here's an example where you need to do a lot of extra work with a closure. It reads a file with '#' style comments, separates comment from non-comment strings, and keeps track of a few counters which a caller in the same file can access as global variables (technically "module-level" variables in Python). comments = 0 uncommented_lines = 0 commented_lines = 0 def tokenize_comments(src): # Point A for line in src: p = line.find('#') if p < 0: yield line #Point B uncommented_lines += 1 else: yield line[:p] #Point C comments += 1 yield line[p:] #Point D commented_lines += 1 #Point E return Try to write a functionally identical implementation of this code with closures that preserves the behavior of the counters. The API could be much improved, for example, we might want to wrap this in a class to get rid of those global variables. But that's not the point. The point is that rewriting functionally equivalent code will require you to keep track of whether the code is at Point A, B, C, or D. I purposely included the counters in this example to make it impossible to unify different control paths, which would make a closure-based implementation a lot easier. For example, if it wasn't for the counters, points B, D, and E would be identical.
- 14y ago
nickporter 14y ago
Unrelated, but I find that the technique used to blur the logo is really cool! He uses two images, one normal and the other blurred. Both images are using fixed positioning. The normal image is set as the `body` background, and the other is set as the background of the content panels (`div.bgover`). So, when you scroll over the panels, the blurred logo is layered over the normal logo, giving it that cool blurred transparency effect.
- webexcess 14y ago
  I stole that from here http://meyerweb.com/eric/css/edge/complexspiral/glassy.html http://meyerweb.com/eric/css/edge/complexspiral/glassy.html
  - nickporter 14y ago
    Very cool, thanks.
- hadem 14y ago
  I found the logo to be incredibly distracting. I left the site nearly immediately after I started to scroll the page.
Marazan 14y ago
That link to Generator Tricks for System Programmers ( http://www.dabeaz.com/generators/index.html http://www.dabeaz.com/generators/index.html ) has been enlightening. For the first time I've totally and fully groked why generators are good at a _practical_ rather than just theoretical level.

171 points by webexcess 14y ago

11001 14y ago

Thank you, great write up! I'd rather see more posts like this on the front page than reviews of the latest tablets.

Another very nice and mind-bending presentation on coroutines, that eventually takes things so far that he creates a mini "OS" with coroutines as tasks, complete with system calls and a scheduler: http://dabeaz.com/coroutines/ http://dabeaz.com/coroutines/

webexcess 14y ago
Thank you, I hadn't seen that. I've added that link to the bottom of my article.
- beagle3 14y ago
  You use "assert gen.next() is None", but asserts might be dropped by optimizations, see e.g. http://docs.python.org/release/2.5.2/ref/assert.html http://docs.python.org/release/2.5.2/ref/assert.html - and your code will break. assert should only be used for debugging, and only for things that should never happen. side effects on assert is a no-no. (Also in C)
manaskarekar 14y ago
The talk by the same David Beazley on Co-routines: http://blip.tv/pycon-us-videos-2009-2010-2011/a-curious-course-on-coroutines-and-concurrency-part-001-2006228 http://blip.tv/pycon-us-videos-2009-2010-2011/a-curious-cour... The talk from David Mertz on Co-routines: https://www.youtube.com/watch?v=b7R3-_ViNxk https://www.youtube.com/watch?v=b7R3-_ViNxk
- mapleoin 14y ago
  Wow that David Beazley presentation is annoying. He spends the first 10 minutes in the introduction, complaining about how he won't have enough time to go through his presentation.
  - d23 14y ago
    I didn't watch the video, but going through the slides I was absolutely fascinated. I've been craving something new and advanced like this out of python for a while.

wookietrader 14y ago

If they could only be pickled...

webexcess 14y ago
That's a feature! Seriously though, not being able to define a serialization method can be a limitation. However, it is always possible to turn a generator into an iterator class with whatever methods you like, but the code may end up looking totally different.
- wookietrader 14y ago
  To me it only looks like a limitation. So yeah, you can write a custom class. But you will need to add a custom "save state" and "load state" in your class' __iter__ method. So all the benefits go away.

pekk 14y ago

These are great. If only one could write async network code with these that wasn't vastly more complicated than the equivalent gevent code.

masklinn 14y ago
Tulip coroutines[0] look rather good. As with C#'s async/await you have to propagate the "asyncness" to the event loop (using `yield from` with a future or another coroutine), which is not necessary in gevent, but that should be the extent of it. [0] http://www.python.org/dev/peps/pep-3156/#coroutines-and-the-scheduler http://www.python.org/dev/peps/pep-3156/#coroutines-and-the-...
- pekk 14y ago
  The implementation described in that PEP is massive. Seems pretty hard to explain...

Xcelerate 14y ago

Somewhat off-topic, but how are generators different than returning an object with a closed over environment? For instance, his first example in CoffeeScript would be: runningAvg = -> count = 0 total = 0 send: (value) -> total += value count += 1 total/count r = runningAvg() console.log r.send 10 console.log r.send 5 console.log r.send 0 console.log r.send 0

deleted 14y ago
[deleted]
csense 14y ago
Generators can restore you to any point in the control flow, whereas your closure-based solution (or an object-oriented solution where variables are class fields) always has to start from the top of the send() function. For simple examples, closures can present an easy alternative implementation. But here's an example where you need to do a lot of extra work with a closure. It reads a file with '#' style comments, separates comment from non-comment strings, and keeps track of a few counters which a caller in the same file can access as global variables (technically "module-level" variables in Python). comments = 0 uncommented_lines = 0 commented_lines = 0 def tokenize_comments(src): # Point A for line in src: p = line.find('#') if p < 0: yield line #Point B uncommented_lines += 1 else: yield line[:p] #Point C comments += 1 yield line[p:] #Point D commented_lines += 1 #Point E return Try to write a functionally identical implementation of this code with closures that preserves the behavior of the counters. The API could be much improved, for example, we might want to wrap this in a class to get rid of those global variables. But that's not the point. The point is that rewriting functionally equivalent code will require you to keep track of whether the code is at Point A, B, C, or D. I purposely included the counters in this example to make it impossible to unify different control paths, which would make a closure-based implementation a lot easier. For example, if it wasn't for the counters, points B, D, and E would be identical.
14y ago

nickporter 14y ago

Unrelated, but I find that the technique used to blur the logo is really cool! He uses two images, one normal and the other blurred. Both images are using fixed positioning. The normal image is set as the `body` background, and the other is set as the background of the content panels (`div.bgover`). So, when you scroll over the panels, the blurred logo is layered over the normal logo, giving it that cool blurred transparency effect.

webexcess 14y ago
I stole that from here http://meyerweb.com/eric/css/edge/complexspiral/glassy.html http://meyerweb.com/eric/css/edge/complexspiral/glassy.html
- nickporter 14y ago
  Very cool, thanks.
hadem 14y ago
I found the logo to be incredibly distracting. I left the site nearly immediately after I started to scroll the page.

Marazan 14y ago

That link to Generator Tricks for System Programmers ( http://www.dabeaz.com/generators/index.html http://www.dabeaz.com/generators/index.html ) has been enlightening. For the first time I've totally and fully groked why generators are good at a _practical_ rather than just theoretical level.