3 ms·

It would be really cool to have the sort of Erlang-style processes in python -- where each process is a green process, so don't pay the cost of OS handling stuf

by devxpy 8y ago

It would be really cool to have the sort of Erlang-style processes in python -- where each process is a green process, so don't pay the cost of OS handling stuff, but at the same time you run multiple interpreters so that you still get the benefit of multiple cores [1]. Implementation-wise, this could be made simpler than removing the GIL, if we just fully give up on the concept of shared memory. (This is above my pay-grade, anyone with CPython internals knowledge, please share your thoughts!) I've been trying to build a shared state system over message passing that doesn't use shared memory at all [2], and I have experienced that OS level processes are a high cost to pay, when you try to do things the Erlang way, launching thousands of processes. [1] https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [2] https://github.com/pycampers/zproc https://github.com/pycampers/zproc

tyingq 8y ago
There's this: http://www.gevent.org/ http://www.gevent.org/
- devxpy 8y ago
  Single core only :(
  - btown 8y ago
    If you need multicore performance, you likely will soon need multi-server performance, in which case you can either use multiprocessing or even containerization to run one gevent “hub” on each core across a cluster. Compared to threads, the memory overhead is minimal.
    - _asummers 8y ago
      Going from single node to multi core doesn't introduce nearly the amount of complexity that making a distributed system does. Recommending that for someone who wants to use more than 1 core on their machine is not a reasonable recommendation. You're not wrong that you might wind up there, at the end, but doing that for the first step is too heavy unless you have a runtime to support those complexities more easily, as in e.g. BEAM.
    - devxpy 8y ago
      Running one gevent hub per core seems viable. Does it support forking? Also I don't think that gevent does preemptive switching. I can already do multi server just fine, but not everything is in the server domain. Also I don't seek multi-core usage for performance, specifically. Versatility is what I'm going for here. An Erlang like system can be both performant and scalable to thousands of clients. The current scope of python really only allows for either one. You can go either multiprocessing, which will give great processing power, but you can only launch so many of them. You can go asyncio route and launch millions of coroutines, but they suffer from performance issues.

by devxpy 8y ago

tyingq 8y ago

There's this: http://www.gevent.org/ http://www.gevent.org/

devxpy 8y ago
Single core only :(
- btown 8y ago
  If you need multicore performance, you likely will soon need multi-server performance, in which case you can either use multiprocessing or even containerization to run one gevent “hub” on each core across a cluster. Compared to threads, the memory overhead is minimal.
  - _asummers 8y ago
    Going from single node to multi core doesn't introduce nearly the amount of complexity that making a distributed system does. Recommending that for someone who wants to use more than 1 core on their machine is not a reasonable recommendation. You're not wrong that you might wind up there, at the end, but doing that for the first step is too heavy unless you have a runtime to support those complexities more easily, as in e.g. BEAM.
  - devxpy 8y ago
    Running one gevent hub per core seems viable. Does it support forking? Also I don't think that gevent does preemptive switching. I can already do multi server just fine, but not everything is in the server domain. Also I don't seek multi-core usage for performance, specifically. Versatility is what I'm going for here. An Erlang like system can be both performant and scalable to thousands of clients. The current scope of python really only allows for either one. You can go either multiprocessing, which will give great processing power, but you can only launch so many of them. You can go asyncio route and launch millions of coroutines, but they suffer from performance issues.