3 ms·

In python we've been having a yield statement for a long time. By far and large, it has not solved the issue of cooperative multitasking. Using it only leads to

by pyalot2 13y ago

In python we've been having a yield statement for a long time. By far and large, it has not solved the issue of cooperative multitasking. Using it only leads to near incomprehensive code where everything becomes an iterable, and every call becomes an iteration. Co-routines solve cooperative multitasking. Which is why they're now being discussed for python in a fashion that doesn't rely on C-stack hacks.

kevingadd 13y ago
A better way to think of this is in terms of degrees of improvement. Going from pure callback passing to yield-based cooperative multitasking increases clarity and removes a lot of opportunities for mistakes, so in practice it ends up being a win - but just like callback passing, you can still hit a point where it's too complicated or you're having to do too much. It's only natural that going from custom task schedulers built on 'yield' to language-native mechanisms for cooperative multitasking improves things further. Even that, though, will likely not 'solve the issue', just make it considerably easier to tackle for individual applications. There's no One True Scheduler or One True Task Primitive, at least as far as I know: everyone seems to want something slightly different.
lucian1900 13y ago
I would say it has almost entirely solved the problem. Twisted with inlineCallbacks makes writing multitasking programs very clear and obvious. Python 3.3 makes it even better by allowing `return` and requiring less wrapping with `yield from`. Implicit stack coroutines (like greenlets) can always be built on top if necessary, but they tend to require horrible things like monkey patching to work together with existing synchronous, blocking code.
- denik 13y ago
  > Implicit stack coroutines (like greenlets) can always be built on top if necessary, but they tend to require horrible things like monkey patching to work together with existing synchronous, blocking code. that's not true. monkey patching is an optional feature that you can use with greenlet-based libraries, but you don't have to. compare that to inlineCallbacks, where you don't have that option.
themckman 13y ago
I recently just had great success using coroutines to build a sort of function pipeline that can work in both code and at the command line with minimal effort; I'm doing a lot of text processing. A contrived example that prints the numbers in range(0, 100) ends up looking something like this: @producer def prod(): for i in range(0,100): yield i @producer @consumer def middleman(): for i in (yield): yield i @consumer def cons(): for i in (yield): print i Producers can yield values and consumers can use yield to get their input. I will fully admit that code, abstracted away by those decorators, is not the prettiest and took a minute to understand when writing. The biggest gotcha was with both a producer and consumer. You have to be very careful to capture the return value of the generator.send(input) call as it ends up being the item of the first yield. Also, pure consumers will immediately throw a StopIteration exception after sending the input. I'm very glad, however, that I stumbled upon using generators in this way as it results in a very clean implementation at this level. The internals are a little dicey, however, which I think is what you're eluding to. I use the functions in multiprocessing.Process objects as the target by first binding them to pipes and also without binding them to anything, in which case they use stdin/stdout.

by pyalot2 13y ago

kevingadd 13y ago

A better way to think of this is in terms of degrees of improvement. Going from pure callback passing to yield-based cooperative multitasking increases clarity and removes a lot of opportunities for mistakes, so in practice it ends up being a win - but just like callback passing, you can still hit a point where it's too complicated or you're having to do too much. It's only natural that going from custom task schedulers built on 'yield' to language-native mechanisms for cooperative multitasking improves things further. Even that, though, will likely not 'solve the issue', just make it considerably easier to tackle for individual applications. There's no One True Scheduler or One True Task Primitive, at least as far as I know: everyone seems to want something slightly different.

lucian1900 13y ago

I would say it has almost entirely solved the problem. Twisted with inlineCallbacks makes writing multitasking programs very clear and obvious. Python 3.3 makes it even better by allowing `return` and requiring less wrapping with `yield from`. Implicit stack coroutines (like greenlets) can always be built on top if necessary, but they tend to require horrible things like monkey patching to work together with existing synchronous, blocking code.

denik 13y ago
> Implicit stack coroutines (like greenlets) can always be built on top if necessary, but they tend to require horrible things like monkey patching to work together with existing synchronous, blocking code. that's not true. monkey patching is an optional feature that you can use with greenlet-based libraries, but you don't have to. compare that to inlineCallbacks, where you don't have that option.

themckman 13y ago

I recently just had great success using coroutines to build a sort of function pipeline that can work in both code and at the command line with minimal effort; I'm doing a lot of text processing. A contrived example that prints the numbers in range(0, 100) ends up looking something like this: @producer def prod(): for i in range(0,100): yield i @producer @consumer def middleman(): for i in (yield): yield i @consumer def cons(): for i in (yield): print i Producers can yield values and consumers can use yield to get their input. I will fully admit that code, abstracted away by those decorators, is not the prettiest and took a minute to understand when writing. The biggest gotcha was with both a producer and consumer. You have to be very careful to capture the return value of the generator.send(input) call as it ends up being the item of the first yield. Also, pure consumers will immediately throw a StopIteration exception after sending the input. I'm very glad, however, that I stumbled upon using generators in this way as it results in a very clean implementation at this level. The internals are a little dicey, however, which I think is what you're eluding to. I use the functions in multiprocessing.Process objects as the target by first binding them to pipes and also without binding them to anything, in which case they use stdin/stdout.