6 ms·

Die Threads: Python async code [video]

59 points by luu 8y ago

dingdingdang 8y ago
Looks like a super nice way of doing threads in Python, here's the rep: https://github.com/dabeaz/thredo https://github.com/dabeaz/thredo Sad to see that it has not received any love for last 4 months, is there another "preferred" thread lib for Python?
- dorfsmay 8y ago
  I use threading and manage my own pool in a list (it really isn't difficult). That's the fastest method I have found.
- jMyles 8y ago
  twisted.internet.threads is always, always my go-to personally. Twisted is an amazing project and has especially shined brightly in the past 3 years or so.
  - richardwhiuk 8y ago
    Twisted doesn't support Python 3, which is problematic for some projects. I've also always found the code it produces difficult to test.
    - jMyles 8y ago
      > Twisted doesn't support Python 3 No no, it now supports Python 3 wonderfully. It took a very long time, because a lot of what Twisted does was more dramatically changed from 2-3 than a lot of projects. Consider, for example, the implications of PEP-3333, which required that headers be in the native str type in Python 2, but also the native str type (ie, unicode) in Python 3. This was a tough problem. [0] However, those days are long behind us. Not only does Twisted support Python 3, but it can interchange its own flow control (ie, the reactor) with the asyncio event loop, and also convert (and ensure) that Futures and Deferreds fire in a way that is cross-compatible. You can also use the inlineCallback decorator to see all of the new coroutine syntactic sugar. Here's a project I'm working on right now that is Python 3.6+ with Twisted: https://github.com/nucypher/nucypher/ https://github.com/nucypher/nucypher/ > I've also always found the code it produces difficult to test. Yeah, I hear that a lot, and for my part, I've just never had that trouble, so I'm not sure how to respond. Have you ever read the "TDD with Twisted" document[1]? Also, the new pytest-twisted tooling helps quite a bit if you're using pytest. 0: https://github.com/twisted/twisted/blob/6ac66416c0238f403a8dc1d42924fb3ba2a2a686/src/twisted/web/wsgi.py#L26 https://github.com/twisted/twisted/blob/6ac66416c0238f403a8d... 1: https://twistedmatrix.com/documents/current/core/howto/trial.html https://twistedmatrix.com/documents/current/core/howto/trial...
      - limaoscarjuliet 8y ago
        BTW, when writing in Twisted use @inlineCallbacks with yields for async code. The code reads as if it was synchronous then. @inlineCallbacks def get_index_status(self, index_id): uri = "{}/index-status?id={}".format(self.SERVER, index_id) #yes, no escaping, as this is silly test only r = yield self.get(uri) returnValue(r.text) Using raw Deferreds is a big more difficult.
      - richardwhiuk 8y ago
        https://twistedmatrix.com/trac/query?status=assigned&status=new&status=reopened&group=status&milestone=Python-3.x https://twistedmatrix.com/trac/query?status=assigned&status=... suggests this isn't done - is that incorrect?
      - deleted 8y ago
        [deleted]
    - limaoscarjuliet 8y ago
      I've pushed Twisted on a 100M US company about 7 years ago and it has continued to work for us big time. The code - tens of thousands of lines in 20+ servers - survived Twisted upgrades for all these years without any changes (OK, there was one related to import, had to add PYTHONPATH=. to env). Kudos to Twisted team!
    - deleted 8y ago
      [deleted]
  - luhn 8y ago
    I haven't used Twisted for a while, what's been happening in the past 3 years?
    - jMyles 8y ago
      I'm not a core developer, so I might be mistaken or missing big things, but from my perspective, the big things are (in addition to the stuff I mentioned above how flow control compatibility): Amber Brown took over as release manager, python 3 compatibility landed, lots of new test utilities, Glyph and some of the original authors took steps back but are still involved, IHandshakeListener (makes TLS easier), support for TLS 1.3 cypher suites (though full support is still pending PyOpenSSL I guess?), WSGIResource supports IPv6... I maintain a Twisted-based web server called hendrix; we have integrated many of the new coolness and are fully Python 3 compatible. Check it out: https://github.com/hendrix/hendrix https://github.com/hendrix/hendrix
- nicolaslem 8y ago
  I've been building a task queue à la Celery or Rq but with threads as a first class citizen: https://github.com/NicolasLM/spinach https://github.com/NicolasLM/spinach
  - coleifer 8y ago
    Huey supports thread, process or greenlet worker models.
- xapata 8y ago
  Not all software needs to change frequently to be useful. https://github.com/dabeaz/thredo/issues/2 https://github.com/dabeaz/thredo/issues/2
- JustSomeNobody 8y ago
  > Sad to see that it has not received any love for last 4 months... Why is this a bad thing? Unless thredo is littered with bugs, who cares how old it is?
  - dingdingdang 8y ago
    Of course, but author is very clear on the early/alpha nature of the project plus there's uncertainty as to whether he will continue this as a project or not.. also he's the only contributor. In other words, if I build a codebase to rely on something other than the stdlib it's nice to know that it'll have at least some support down the line.
- Waterluvian 8y ago
  The stdlib threading and concurrent.futures library goes a long way. It's still up to the developer to understand threading well enough before digging in. I'm not sure any library can eliminate that need. I also use gevent a lot. I'm currently in love with cooperative multitasking.
rerx 8y ago
I missed some discussion of how this is built on top of async/await. Edit: Here is a recording of the talk where you can both see the slides and the speaker: https://www.youtube.com/watch?v=xOyJiN3yGfU https://www.youtube.com/watch?v=xOyJiN3yGfU
- erdewit 8y ago
  It's built on top of threading really (imported via curio).
devxpy 8y ago
It would be really cool to have the sort of Erlang-style processes in python -- where each process is a green process, so don't pay the cost of OS handling stuff, but at the same time you run multiple interpreters so that you still get the benefit of multiple cores [1]. Implementation-wise, this could be made simpler than removing the GIL, if we just fully give up on the concept of shared memory. (This is above my pay-grade, anyone with CPython internals knowledge, please share your thoughts!) I've been trying to build a shared state system over message passing that doesn't use shared memory at all [2], and I have experienced that OS level processes are a high cost to pay, when you try to do things the Erlang way, launching thousands of processes. [1] https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [2] https://github.com/pycampers/zproc https://github.com/pycampers/zproc
- tyingq 8y ago
  There's this: http://www.gevent.org/ http://www.gevent.org/
  - devxpy 8y ago
    Single core only :(
    - btown 8y ago
      If you need multicore performance, you likely will soon need multi-server performance, in which case you can either use multiprocessing or even containerization to run one gevent “hub” on each core across a cluster. Compared to threads, the memory overhead is minimal.
      - _asummers 8y ago
        Going from single node to multi core doesn't introduce nearly the amount of complexity that making a distributed system does. Recommending that for someone who wants to use more than 1 core on their machine is not a reasonable recommendation. You're not wrong that you might wind up there, at the end, but doing that for the first step is too heavy unless you have a runtime to support those complexities more easily, as in e.g. BEAM.
miduil 8y ago
In case you are wondering, the editor used at this talk is called "Mu" and got previously discussed on HN about three months ago [0]. [0]: https://news.ycombinator.com/item?id=17638067 https://news.ycombinator.com/item?id=17638067

59 points by luu 8y ago

Looks like a super nice way of doing threads in Python, here's the rep: https://github.com/dabeaz/thredo https://github.com/dabeaz/thredo Sad to see that it has not received any love for last 4 months, is there another "preferred" thread lib for Python?

dorfsmay 8y ago
I use threading and manage my own pool in a list (it really isn't difficult). That's the fastest method I have found.
jMyles 8y ago
twisted.internet.threads is always, always my go-to personally. Twisted is an amazing project and has especially shined brightly in the past 3 years or so.
- richardwhiuk 8y ago
  Twisted doesn't support Python 3, which is problematic for some projects. I've also always found the code it produces difficult to test.
  - jMyles 8y ago
    > Twisted doesn't support Python 3 No no, it now supports Python 3 wonderfully. It took a very long time, because a lot of what Twisted does was more dramatically changed from 2-3 than a lot of projects. Consider, for example, the implications of PEP-3333, which required that headers be in the native str type in Python 2, but also the native str type (ie, unicode) in Python 3. This was a tough problem. [0] However, those days are long behind us. Not only does Twisted support Python 3, but it can interchange its own flow control (ie, the reactor) with the asyncio event loop, and also convert (and ensure) that Futures and Deferreds fire in a way that is cross-compatible. You can also use the inlineCallback decorator to see all of the new coroutine syntactic sugar. Here's a project I'm working on right now that is Python 3.6+ with Twisted: https://github.com/nucypher/nucypher/ https://github.com/nucypher/nucypher/ > I've also always found the code it produces difficult to test. Yeah, I hear that a lot, and for my part, I've just never had that trouble, so I'm not sure how to respond. Have you ever read the "TDD with Twisted" document[1]? Also, the new pytest-twisted tooling helps quite a bit if you're using pytest. 0: https://github.com/twisted/twisted/blob/6ac66416c0238f403a8dc1d42924fb3ba2a2a686/src/twisted/web/wsgi.py#L26 https://github.com/twisted/twisted/blob/6ac66416c0238f403a8d... 1: https://twistedmatrix.com/documents/current/core/howto/trial.html https://twistedmatrix.com/documents/current/core/howto/trial...
    - limaoscarjuliet 8y ago
      BTW, when writing in Twisted use @inlineCallbacks with yields for async code. The code reads as if it was synchronous then. @inlineCallbacks def get_index_status(self, index_id): uri = "{}/index-status?id={}".format(self.SERVER, index_id) #yes, no escaping, as this is silly test only r = yield self.get(uri) returnValue(r.text) Using raw Deferreds is a big more difficult.
    - richardwhiuk 8y ago
      https://twistedmatrix.com/trac/query?status=assigned&status=new&status=reopened&group=status&milestone=Python-3.x https://twistedmatrix.com/trac/query?status=assigned&status=... suggests this isn't done - is that incorrect?
    - deleted 8y ago
      [deleted]
  - limaoscarjuliet 8y ago
    I've pushed Twisted on a 100M US company about 7 years ago and it has continued to work for us big time. The code - tens of thousands of lines in 20+ servers - survived Twisted upgrades for all these years without any changes (OK, there was one related to import, had to add PYTHONPATH=. to env). Kudos to Twisted team!
  - deleted 8y ago
    [deleted]
- luhn 8y ago
  I haven't used Twisted for a while, what's been happening in the past 3 years?
  - jMyles 8y ago
    I'm not a core developer, so I might be mistaken or missing big things, but from my perspective, the big things are (in addition to the stuff I mentioned above how flow control compatibility): Amber Brown took over as release manager, python 3 compatibility landed, lots of new test utilities, Glyph and some of the original authors took steps back but are still involved, IHandshakeListener (makes TLS easier), support for TLS 1.3 cypher suites (though full support is still pending PyOpenSSL I guess?), WSGIResource supports IPv6... I maintain a Twisted-based web server called hendrix; we have integrated many of the new coolness and are fully Python 3 compatible. Check it out: https://github.com/hendrix/hendrix https://github.com/hendrix/hendrix
nicolaslem 8y ago
I've been building a task queue à la Celery or Rq but with threads as a first class citizen: https://github.com/NicolasLM/spinach https://github.com/NicolasLM/spinach
- coleifer 8y ago
  Huey supports thread, process or greenlet worker models.
xapata 8y ago
Not all software needs to change frequently to be useful. https://github.com/dabeaz/thredo/issues/2 https://github.com/dabeaz/thredo/issues/2
JustSomeNobody 8y ago
> Sad to see that it has not received any love for last 4 months... Why is this a bad thing? Unless thredo is littered with bugs, who cares how old it is?
- dingdingdang 8y ago
  Of course, but author is very clear on the early/alpha nature of the project plus there's uncertainty as to whether he will continue this as a project or not.. also he's the only contributor. In other words, if I build a codebase to rely on something other than the stdlib it's nice to know that it'll have at least some support down the line.
Waterluvian 8y ago
The stdlib threading and concurrent.futures library goes a long way. It's still up to the developer to understand threading well enough before digging in. I'm not sure any library can eliminate that need. I also use gevent a lot. I'm currently in love with cooperative multitasking.

rerx 8y ago

I missed some discussion of how this is built on top of async/await. Edit: Here is a recording of the talk where you can both see the slides and the speaker: https://www.youtube.com/watch?v=xOyJiN3yGfU https://www.youtube.com/watch?v=xOyJiN3yGfU

erdewit 8y ago
It's built on top of threading really (imported via curio).

devxpy 8y ago

It would be really cool to have the sort of Erlang-style processes in python -- where each process is a green process, so don't pay the cost of OS handling stuff, but at the same time you run multiple interpreters so that you still get the benefit of multiple cores [1]. Implementation-wise, this could be made simpler than removing the GIL, if we just fully give up on the concept of shared memory. (This is above my pay-grade, anyone with CPython internals knowledge, please share your thoughts!) I've been trying to build a shared state system over message passing that doesn't use shared memory at all [2], and I have experienced that OS level processes are a high cost to pay, when you try to do things the Erlang way, launching thousands of processes. [1] https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [2] https://github.com/pycampers/zproc https://github.com/pycampers/zproc

tyingq 8y ago
There's this: http://www.gevent.org/ http://www.gevent.org/
- devxpy 8y ago
  Single core only :(
  - btown 8y ago
    If you need multicore performance, you likely will soon need multi-server performance, in which case you can either use multiprocessing or even containerization to run one gevent “hub” on each core across a cluster. Compared to threads, the memory overhead is minimal.
    - _asummers 8y ago
      Going from single node to multi core doesn't introduce nearly the amount of complexity that making a distributed system does. Recommending that for someone who wants to use more than 1 core on their machine is not a reasonable recommendation. You're not wrong that you might wind up there, at the end, but doing that for the first step is too heavy unless you have a runtime to support those complexities more easily, as in e.g. BEAM.

miduil 8y ago

In case you are wondering, the editor used at this talk is called "Mu" and got previously discussed on HN about three months ago [0]. [0]: https://news.ycombinator.com/item?id=17638067 https://news.ycombinator.com/item?id=17638067