5 ms·

The general sentiment in this thread seems to be that Guido et al "blessed" the Twisted way, it's a pity and now using asyncio and friends is the only way to su

by plq 10y ago

The general sentiment in this thread seems to be that Guido et al "blessed" the Twisted way, it's a pity and now using asyncio and friends is the only way to suspend a Python stack in a concurrent program besides using OS-level threads. This is not correct. Green threads, as a programming paradigm, are just a sub-optimal (but cheaper) way of doing preemptive multitasking. Yes, switch is explicit, but user code can't know what will switch. So you are not supposed to treat green threads any differently from OS threads. ie you still need your mutexes, semaphores etc. if you want to avoid race conditions. Again, that's because the caller doesn't know whether a function will yield execution to the next queued task at some point. Guido explains this point pretty well in one of his PyCon keynotes. So green threads may be great but they don't bring anything new to the table. However, Twisted-style concurrency (aka cooperative multitasking) is a different paradigm. In Twisted, you know that you only have one thread running at a time, so you actually don't need any thread synchronization primitives when accessing non-local state. This simplifies things a great deal. Yes, not having to spawn a thread for every single concurrent IO operation has other great benefits, but that's not the reason why CPython now has a blessed event loop -- it's cooperative-multitasking-the-paradigm. Before asyncio, there was no standard way of doing cooperative multitasking. Now there is and it's baked right into the language. Use it if you like it. If not, the old ways of doing things work just fine in Python 3. I'll admit that the concurrency model in Python 3.4 was not perfect. However, what we have with Python 3.5 and up looks quite polished.

int_19h 10y ago
Another issue with green threads is that they usually do not work very well (or at all) across API, much less ABI, boundaries. Basically, the moment your control flow is calling into some shared library, you probably want C API at that boundary for the sake of portability and stability. Exposing what is an, essentially, promise-based system that way is not hard. Better yet, if the other side has some analogous construct, you can map to that. But how do you do it with green threads? Even platforms that have an OS-wide unified green thread primitive, like Win32 fibers, which would presumably solve this problem, find their use very lackluster, because many languages and frameworks just plain don't support fibers correctly. Even CLR tried to do it once and dropped the feature; forget about Python etc.
fnl 10y ago
Maybe the real issue with Twisted/asyncio is that it requires that all your code is "asyncio-ready". A bit like lock-free programming and only using linked lists. That said, it's great that Python has something like this in its stdlib now, a bit like Node, Akka, and Go. But maybe asyncio needs a great big warning sign: "Use this only if you know why you need it and understand the impact it will have on your application's architecture."
- rlpb 10y ago
  > Maybe the real issue with Twisted/asyncio is that it requires that all your code is "asyncio-ready". I don't think this is true. It's easy enough to ask for something that isn't "asyncio-ready" to run in a real thread. Give it a function call. It will run it in a thread, and give you a Future for when it's done. See https://docs.python.org/3/library/asyncio-eventloop.html#executor https://docs.python.org/3/library/asyncio-eventloop.html#exe... for details. Sure, it's a pain because you don't get the benefit that asyncio gives you for that part of code, but isn't at all an "all or nothing" proposition.
rdtsc 10y ago
> ie you still need your mutexes, semaphores etc. if you want to avoid race conditions [...] In Twisted, you know that you only have one thread running at a time, so you actually don't need any thread synchronization primitives when accessing non-local state. I disagree (unless you have a toy example or demo). As the software grows it becomes peppered with yields at the top level. Everything start yield -- authentication is an yield, launching a background job is an yield, writing to the databases. At that point you might find that some shared data has to be protected as well from concurrent IO requests so you still need semaphores and locks. Heck, Twisted has http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.DeferredSemaphore.html http://twistedmatrix.com/documents/9.0.0/api/twisted.interne... I had to use it too because multiple callback chains modifying and accessing the same state had a race condition. Yeah I knew I could multiply a matrix quickly without having to acquire a lock, but I wasn't multiplying matrices I doing IO-bound things. With concurrent IO requests there is still a potential for a data race. > So green threads may be great but they don't bring anything new to the table. Green threads bring: 1. Lighter weight concurrency units than native threads. 2. Green threads don't fragment the library ecosystem. (For a language with batteries included this is rough). If you have been using Twisted you know what I am talking about. "Oh I found a library that does this protocol. Ah, but it is not Twisted, can't work with it. Start writing a parallel implementation 3. Provide a better abstraction without extra code bloat. When you really want to put an item in a shopping cart, do you really care anything underneath yields? You want to write : authenticate(); get_price(); get_availability(); update_cart(); respond_to_user(); and such. That code should not know about select loops and reactor and awaits. Lower level frameworks should handle that and top level code should be clean and obvious. After switching from Twisted (even with inlineCallbacks) I cut the total lines of code in a large code base by half by using eventlet (that was before gevent), because it cut all the callbacks and handlers and all that stuff. Those are lines of code cluttering the business logic, they need maintenance, they need people to read them when bugs happen. Are Gevent and Eventlet ideal? No, they have been always a hack. But in practice I'll take the monkey-patching vs awaits, yield or deferreds and having to hunt for or rewrite libraries which speak that particular IO "language". I understand that on paper and in small example those look neat in clean, in practice it turns into a mess.
- plq 10y ago
danappelxx 10y ago
Have you seen libmill[0]? Single-threaded coroutines - no mutexes, semaphores, etc. We're using it in Zewo[1] with great success. [0]: http://libmill.org http://libmill.org [1]: https://github.com/Zewo/Zewo https://github.com/Zewo/Zewo
ntoshev 10y ago
I disagree that the code can't know what will switch with green threads. They switch on I/O and when explicitly requested. An example of using this to get rid of concurrency control: http://www.underengineering.com/2014/05/22/DIY-NoSql/ http://www.underengineering.com/2014/05/22/DIY-NoSql/ If you want to make sure that there are no context switching in certain part of your code, you can do it ASSERT-style, something like: # enable this only if you use atomic, so a new module that should be imported before gevent in_transaction = False if __debug__: import greenlet old_switch=greenlet.greenlet.switch class _greenlet(greenlet.greenlet): def switch(*args, **kwargs): if in_transaction: raise Exception('Switching context during / atomic') old_switch(*args, **kwargs) setattr(greenlet, 'greenlet', _greenlet) @contextmanager def atomic(): """ Ensure that a function or a block of code is atomic, raise exception if it's not Usage: @atomic() def myTransaction(...): ... or with atomic(): ... """ global in_transaction in_transaction = True yield in_transaction = False
- plq 10y ago
  And how is this not a mutex?
  - deleted 10y ago
    [deleted]
  - ntoshev 10y ago
    This only runs in the debug version of the code. I hope your mutexes run in production, too.
    - lmm 10y ago
      So you rely on hoping your tests exercise all possible paths, because if not you get silent race conditions in prod?

by plq 10y ago

int_19h 10y ago

Another issue with green threads is that they usually do not work very well (or at all) across API, much less ABI, boundaries. Basically, the moment your control flow is calling into some shared library, you probably want C API at that boundary for the sake of portability and stability. Exposing what is an, essentially, promise-based system that way is not hard. Better yet, if the other side has some analogous construct, you can map to that. But how do you do it with green threads? Even platforms that have an OS-wide unified green thread primitive, like Win32 fibers, which would presumably solve this problem, find their use very lackluster, because many languages and frameworks just plain don't support fibers correctly. Even CLR tried to do it once and dropped the feature; forget about Python etc.

fnl 10y ago

Maybe the real issue with Twisted/asyncio is that it requires that all your code is "asyncio-ready". A bit like lock-free programming and only using linked lists. That said, it's great that Python has something like this in its stdlib now, a bit like Node, Akka, and Go. But maybe asyncio needs a great big warning sign: "Use this only if you know why you need it and understand the impact it will have on your application's architecture."

rlpb 10y ago
> Maybe the real issue with Twisted/asyncio is that it requires that all your code is "asyncio-ready". I don't think this is true. It's easy enough to ask for something that isn't "asyncio-ready" to run in a real thread. Give it a function call. It will run it in a thread, and give you a Future for when it's done. See https://docs.python.org/3/library/asyncio-eventloop.html#executor https://docs.python.org/3/library/asyncio-eventloop.html#exe... for details. Sure, it's a pain because you don't get the benefit that asyncio gives you for that part of code, but isn't at all an "all or nothing" proposition.

rdtsc 10y ago

> ie you still need your mutexes, semaphores etc. if you want to avoid race conditions [...] In Twisted, you know that you only have one thread running at a time, so you actually don't need any thread synchronization primitives when accessing non-local state. I disagree (unless you have a toy example or demo). As the software grows it becomes peppered with yields at the top level. Everything start yield -- authentication is an yield, launching a background job is an yield, writing to the databases. At that point you might find that some shared data has to be protected as well from concurrent IO requests so you still need semaphores and locks. Heck, Twisted has http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.DeferredSemaphore.html http://twistedmatrix.com/documents/9.0.0/api/twisted.interne... I had to use it too because multiple callback chains modifying and accessing the same state had a race condition. Yeah I knew I could multiply a matrix quickly without having to acquire a lock, but I wasn't multiplying matrices I doing IO-bound things. With concurrent IO requests there is still a potential for a data race. > So green threads may be great but they don't bring anything new to the table. Green threads bring: 1. Lighter weight concurrency units than native threads. 2. Green threads don't fragment the library ecosystem. (For a language with batteries included this is rough). If you have been using Twisted you know what I am talking about. "Oh I found a library that does this protocol. Ah, but it is not Twisted, can't work with it. Start writing a parallel implementation 3. Provide a better abstraction without extra code bloat. When you really want to put an item in a shopping cart, do you really care anything underneath yields? You want to write : authenticate(); get_price(); get_availability(); update_cart(); respond_to_user(); and such. That code should not know about select loops and reactor and awaits. Lower level frameworks should handle that and top level code should be clean and obvious. After switching from Twisted (even with inlineCallbacks) I cut the total lines of code in a large code base by half by using eventlet (that was before gevent), because it cut all the callbacks and handlers and all that stuff. Those are lines of code cluttering the business logic, they need maintenance, they need people to read them when bugs happen. Are Gevent and Eventlet ideal? No, they have been always a hack. But in practice I'll take the monkey-patching vs awaits, yield or deferreds and having to hunt for or rewrite libraries which speak that particular IO "language". I understand that on paper and in small example those look neat in clean, in practice it turns into a mess.

plq 10y ago

danappelxx 10y ago

Have you seen libmill[0]? Single-threaded coroutines - no mutexes, semaphores, etc. We're using it in Zewo[1] with great success. [0]: http://libmill.org http://libmill.org [1]: https://github.com/Zewo/Zewo https://github.com/Zewo/Zewo

ntoshev 10y ago

I disagree that the code can't know what will switch with green threads. They switch on I/O and when explicitly requested. An example of using this to get rid of concurrency control: http://www.underengineering.com/2014/05/22/DIY-NoSql/ http://www.underengineering.com/2014/05/22/DIY-NoSql/ If you want to make sure that there are no context switching in certain part of your code, you can do it ASSERT-style, something like: # enable this only if you use atomic, so a new module that should be imported before gevent in_transaction = False if __debug__: import greenlet old_switch=greenlet.greenlet.switch class _greenlet(greenlet.greenlet): def switch(*args, **kwargs): if in_transaction: raise Exception('Switching context during / atomic') old_switch(*args, **kwargs) setattr(greenlet, 'greenlet', _greenlet) @contextmanager def atomic(): """ Ensure that a function or a block of code is atomic, raise exception if it's not Usage: @atomic() def myTransaction(...): ... or with atomic(): ... """ global in_transaction in_transaction = True yield in_transaction = False

plq 10y ago
And how is this not a mutex?
- deleted 10y ago
  [deleted]
- ntoshev 10y ago
  This only runs in the debug version of the code. I hope your mutexes run in production, too.
  - lmm 10y ago
    So you rely on hoping your tests exercise all possible paths, because if not you get silent race conditions in prod?