3 ms·

Coroutines + a scheduler become an awesome replacement for threads. They're lighter weight and don't thrash the GIL the same way Python threads do. Greenlet is

by j2labs 15y ago

Coroutines + a scheduler become an awesome replacement for threads. They're lighter weight and don't thrash the GIL the same way Python threads do. Greenlet is an optimized implementation of coroutines for Python. Gevent or Eventlet combine greenlet + a scheduler and even monkeypatch your blocking calls into nonblocking ones. I use eventlet for my load testing tool, Micro Army, to create nonblocking SSH pools. I can SSH to 2 boxes and run config scripts in parallel in 58 seconds, but I can SSH to 100 boxes and run the same tasks in 106 seconds. The code couldn't be simpler too. Here's the proof. SSH details for a single host: https://github.com/j2labs/microarmy/blob/master/microarmy/firepower.py#L149 https://github.com/j2labs/microarmy/blob/master/microarmy/fi... SSH to every host in a list: https://github.com/j2labs/microarmy/blob/master/microarmy/firepower.py#L188 https://github.com/j2labs/microarmy/blob/master/microarmy/fi... Notice that the code is basically just spawning a coroutine and iterating across a list of coroutines. Done. No callback spaghetti, yet all of this is async and nonblocking.

pault 15y ago
Very interesting. Does Greenlet use threads behind the scenes? Do you run into the same problems with deadlocks, debugging, etc?
- anonymoushn 15y ago
  It's single-threaded. I haven't looked under the hood, but I would expect it to work by copying stacks around. In my experience it's been very easy to use.
fredliu 15y ago
ditto for "coroutine + scheduler as replacement of threads". Another example using this framework is Simpy (http://simpy.sourceforge.net/ http://simpy.sourceforge.net/), a process based discrete event simulation platform. You program in a way similar to the traditional multi-process/thread manner, but don't have to worry about mutex and locks at all, very sweet and productive! The only caveat for this framework is that it's hard to fully utilize the power of multi-core systems, since it's single threaded. But if your application is more I/O bound than CPU bound, this framework could potentially give you huge benefit of multi-threading w/o the complexity.

by j2labs 15y ago

pault 15y ago

Very interesting. Does Greenlet use threads behind the scenes? Do you run into the same problems with deadlocks, debugging, etc?

anonymoushn 15y ago
It's single-threaded. I haven't looked under the hood, but I would expect it to work by copying stacks around. In my experience it's been very easy to use.

fredliu 15y ago

ditto for "coroutine + scheduler as replacement of threads". Another example using this framework is Simpy (http://simpy.sourceforge.net/ http://simpy.sourceforge.net/), a process based discrete event simulation platform. You program in a way similar to the traditional multi-process/thread manner, but don't have to worry about mutex and locks at all, very sweet and productive! The only caveat for this framework is that it's hard to fully utilize the power of multi-core systems, since it's single threaded. But if your application is more I/O bound than CPU bound, this framework could potentially give you huge benefit of multi-threading w/o the complexity.