4 ms·

I find generators powerful and I use them all the time, but I haven't found a use for coroutines in my code yet. Maybe I don't grok them fully.

by atdt 15y ago

jflatow 15y ago
I feel the same way. I've found excuses to use relatively 'obscure' Python features (e.g. metaclasses) in the past, but for some reason coroutines have never seemed like the right solution to any problem I have faced. This might be because I don't fully understand them, or perhaps I am using them all the time and have no idea. But if that's the case I think that means they are either too difficult a concept to be of practical use, or an unintuitive way of thinking about what they are actually doing.
j2labs 15y ago
Coroutines + a scheduler become an awesome replacement for threads. They're lighter weight and don't thrash the GIL the same way Python threads do. Greenlet is an optimized implementation of coroutines for Python. Gevent or Eventlet combine greenlet + a scheduler and even monkeypatch your blocking calls into nonblocking ones. I use eventlet for my load testing tool, Micro Army, to create nonblocking SSH pools. I can SSH to 2 boxes and run config scripts in parallel in 58 seconds, but I can SSH to 100 boxes and run the same tasks in 106 seconds. The code couldn't be simpler too. Here's the proof. SSH details for a single host: https://github.com/j2labs/microarmy/blob/master/microarmy/firepower.py#L149 https://github.com/j2labs/microarmy/blob/master/microarmy/fi... SSH to every host in a list: https://github.com/j2labs/microarmy/blob/master/microarmy/firepower.py#L188 https://github.com/j2labs/microarmy/blob/master/microarmy/fi... Notice that the code is basically just spawning a coroutine and iterating across a list of coroutines. Done. No callback spaghetti, yet all of this is async and nonblocking.
- pault 15y ago
  Very interesting. Does Greenlet use threads behind the scenes? Do you run into the same problems with deadlocks, debugging, etc?
  - anonymoushn 15y ago
    It's single-threaded. I haven't looked under the hood, but I would expect it to work by copying stacks around. In my experience it's been very easy to use.
- fredliu 15y ago
  ditto for "coroutine + scheduler as replacement of threads". Another example using this framework is Simpy (http://simpy.sourceforge.net/ http://simpy.sourceforge.net/), a process based discrete event simulation platform. You program in a way similar to the traditional multi-process/thread manner, but don't have to worry about mutex and locks at all, very sweet and productive! The only caveat for this framework is that it's hard to fully utilize the power of multi-core systems, since it's single threaded. But if your application is more I/O bound than CPU bound, this framework could potentially give you huge benefit of multi-threading w/o the complexity.

I find generators powerful and I use them all the time, but I haven't found a use for coroutines in my code yet. Maybe I don't grok them fully.

by atdt 15y ago

I find generators powerful and I use them all the time, but I haven't found a use for coroutines in my code yet. Maybe I don't grok them fully.

jflatow 15y ago

I feel the same way. I've found excuses to use relatively 'obscure' Python features (e.g. metaclasses) in the past, but for some reason coroutines have never seemed like the right solution to any problem I have faced. This might be because I don't fully understand them, or perhaps I am using them all the time and have no idea. But if that's the case I think that means they are either too difficult a concept to be of practical use, or an unintuitive way of thinking about what they are actually doing.

j2labs 15y ago

Coroutines + a scheduler become an awesome replacement for threads. They're lighter weight and don't thrash the GIL the same way Python threads do. Greenlet is an optimized implementation of coroutines for Python. Gevent or Eventlet combine greenlet + a scheduler and even monkeypatch your blocking calls into nonblocking ones. I use eventlet for my load testing tool, Micro Army, to create nonblocking SSH pools. I can SSH to 2 boxes and run config scripts in parallel in 58 seconds, but I can SSH to 100 boxes and run the same tasks in 106 seconds. The code couldn't be simpler too. Here's the proof. SSH details for a single host: https://github.com/j2labs/microarmy/blob/master/microarmy/firepower.py#L149 https://github.com/j2labs/microarmy/blob/master/microarmy/fi... SSH to every host in a list: https://github.com/j2labs/microarmy/blob/master/microarmy/firepower.py#L188 https://github.com/j2labs/microarmy/blob/master/microarmy/fi... Notice that the code is basically just spawning a coroutine and iterating across a list of coroutines. Done. No callback spaghetti, yet all of this is async and nonblocking.

pault 15y ago
Very interesting. Does Greenlet use threads behind the scenes? Do you run into the same problems with deadlocks, debugging, etc?
- anonymoushn 15y ago
  It's single-threaded. I haven't looked under the hood, but I would expect it to work by copying stacks around. In my experience it's been very easy to use.
fredliu 15y ago
ditto for "coroutine + scheduler as replacement of threads". Another example using this framework is Simpy (http://simpy.sourceforge.net/ http://simpy.sourceforge.net/), a process based discrete event simulation platform. You program in a way similar to the traditional multi-process/thread manner, but don't have to worry about mutex and locks at all, very sweet and productive! The only caveat for this framework is that it's hard to fully utilize the power of multi-core systems, since it's single threaded. But if your application is more I/O bound than CPU bound, this framework could potentially give you huge benefit of multi-threading w/o the complexity.