3 ms·

How come there's no work stealing? Green threads are supposed to be backed by some N:M thread pool, no? Also, isn't the problem that JSON decoding (or whatever

by pas 7y ago

How come there's no work stealing? Green threads are supposed to be backed by some N:M thread pool, no? Also, isn't the problem that JSON decoding (or whatever computation) simply block the thread and the other green threads cannot proceed at all, because there are simply no safepoints (yield points) inside these low level functions? And in all these cases shouldn't the application estimate work (eg. in case of JSON if the string is longer than 100K), and if it's too big just put it on a dedicated heavy compute N:N thread pool? For Python it's best practice anyway because the GIL, no?

ryanobjc 7y ago
Is there a n:m green threading mode for python? What does gunicorn/gevent do? Sounds like “not that”. Yes best practices and all that, but that really doesn’t sound like what’s happening.
- Matthias247 7y ago
  I don’t know what gevent is doing. But if you have a global interpreter lock then M:N might not be worth it, since only thread will make progress (outside of the syscalls, which are non blocking).
- pas 7y ago
  Python was always more about breadth than depth. (CPython is full of known inefficiencies, but it's with us since 1989, and basically the core dev who worked most on performance - Victor Stinner - thinks the best way is to introduce subinterpreters - https://github.com/vstinner/talks/blob/master/2019-EuroPython/python_performance.pdf https://github.com/vstinner/talks/blob/master/2019-EuroPytho... ) Oh, that PDF is interesting, Python 3.8 has shared memory for multiprocessing, no more pipe objects between processes. Furthermore extension and internal stuff always had the ability to release the GIL and do its own thing (for example, on a threadpool, or using async/nonblocking I/O). But I have no idea about Gevent. I never liked it. (Just as Twisted/Tornado it was too much magic for too little benefit.)

by pas 7y ago

ryanobjc 7y ago

Is there a n:m green threading mode for python? What does gunicorn/gevent do? Sounds like “not that”. Yes best practices and all that, but that really doesn’t sound like what’s happening.

Matthias247 7y ago
I don’t know what gevent is doing. But if you have a global interpreter lock then M:N might not be worth it, since only thread will make progress (outside of the syscalls, which are non blocking).
pas 7y ago
Python was always more about breadth than depth. (CPython is full of known inefficiencies, but it's with us since 1989, and basically the core dev who worked most on performance - Victor Stinner - thinks the best way is to introduce subinterpreters - https://github.com/vstinner/talks/blob/master/2019-EuroPython/python_performance.pdf https://github.com/vstinner/talks/blob/master/2019-EuroPytho... ) Oh, that PDF is interesting, Python 3.8 has shared memory for multiprocessing, no more pipe objects between processes. Furthermore extension and internal stuff always had the ability to release the GIL and do its own thing (for example, on a threadpool, or using async/nonblocking I/O). But I have no idea about Gevent. I never liked it. (Just as Twisted/Tornado it was too much magic for too little benefit.)