6 ms·

In my opinion, Python needs to have an urgent answer to the use of multiple cores. AMD just launched a CPU with 96 cores. Today the use on multiple cores is thr

by bratao 3y ago

In my opinion, Python needs to have an urgent answer to the use of multiple cores. AMD just launched a CPU with 96 cores. Today the use on multiple cores is through the use multiprocess, which has many limitations. I understand that the multiple interpreters could come with something like Goroutines, but I still like the real multi thread option more.

brianwawok 3y ago
My server can easily fill all 96 running standard Python under gunicorn.
- NegativeLatency 3y ago
  and if you use and async framework it's even more capable
- iterateoften 3y ago
  gunicorn uses multiple processes
  - aeyes 3y ago
    You can still share most of the memory if you load everything before the fork so why is having multiple processes a problem?
    - rtpg 3y ago
      Ive found in practice that memory sharing doesn’t work as much as one would want, and multiple processes cost hundreds of megs (annoying!)
    - skrause 3y ago
      One operating system I have to deploy on (Windows) has neither fork() nor gunicorn.
    - dontlaugh 3y ago
      Reference counting prevents that from working well. Very little sharing actually happens by default.
      - sitkack 3y ago
        If the counts were kept somewhere else, they wouldn't pollute, or if you could turn off ref counts for everything allocated before the call to fork, then you would also be good. The other option is you could allocate all the shared data off of the Python heap prior to the fork call.
        dontlaugh 3y ago
        Instagram have a system for making some objects immortal before the fork, but it’s not the default behaviour in Python. It’s also not easy to set up in some cases.
        aeyes 3y ago
        PEP683 finally landed in Python 3.12: https://peps.python.org/pep-0683/ https://peps.python.org/pep-0683/ It's been in Pyston for a while.
- b5n 3y ago
  Even simpler: parallel -j96 'python -c "print('{}')"' ::: $(for i in {1..96};do echo $i;done)
dekhn 3y ago
I've changed my opinion about this after many years. A long time ago IronPython was released which showed that you could build a high performance Python interpreter that was GIL-free. It had thread-safe containers (so multiple threads could work against the same lists, dictionaries, etc) and in some cases was faster than CPython (it was implemented using CLR and .net) When I saw IronPython I was immediately convinced that CPython should be the same way- already low-cost dual and quad core Intel machines were becoming available and it seemed like core counts were going to increase faster than clock rates. I figured that a small hit to serial performance would be more than acceptable if people could write multithreaded systems in Python, in much the same way as I wrote multithreaded C++. Over time after watching nogil not going anywhere in CPython (the python leadership didn't want to do nogil), with concomitant speedups in the single-processor implementation, along with the increasing use of C++ code that releases the GIL, and seeing that many people just weren't good at multithreaded programming, I have started to conclude that the multithreading/multiprocess in Python today is about the best we can get. That is, instead of having threaded containers and multiple interpreter threads all banging on the same underlying data, it's a lot easier to just use threads as work queues that have minimal interaction with other threads. So that's where I've ended up: some of my code using multiprocessing, typically the Pool or ThreadPool, with the concurrent future API to handle result-gathering, barriers, etc. Other code has an external system that starts many python processes from the command line and waits for those processes to complete. Other code is single-threaded in python and launches C++ cores that launch multiple threads just to return a computational result to python faster. And I think trying to do both gil and nogil interpreters, rather than committing to one or the other, the python leadership will sign us up for untold inconveniences around packaging. We already see this in the move to async and it will only be worse with threading. So sad to say I think sticking to GIL and using the approaches I mentioned above (along with others that work well for concurrent, rather than parallel, computing, like coroutines) is the best thing to do right now and I'm a bit bummed that the leadership signalled their intent to accomodate both.
- alfalfasprout 3y ago
  You've been able to release GIL for a long time though in the C ABI (and C++ via pybind) and that hasn't removed the need for nogil. The reason is that a single thread can invoke multicore c/c++ code (even calling into specialized accelerators if needed) but having python objects that are shared between threads is extremely clunky. And multiprocessing results in a lot of communication overhead. Worse-- python code cannot proceed in another thread. Mixing python and C++ is very common in ML and scientific computing workloads. > And I think trying to do both gil and nogil interpreters, rather than committing to one or the other, the python leadership will sign us up for untold inconveniences around packaging. We already see this in the move to async and it will only be worse with threading. How has packaging been affected by async? As long as you have a compatible python version what issue do you run into?
jshen 3y ago
Why? What are the use cases? I honestly can't think of any. If you are making a web app, you can use those cores just fine. If you are doing ML you will be calling into native code which can also do it just fine. If you are trying to make a AAA game, you shouldn't use python, etc. Not sure what the use case is.
- cpgxiii 3y ago
  > If you are doing ML you will be calling into native code which can also do it just fine. This is exactly the use case. You can only parallelize in the native code if the boundary between Python and native code is absolute. But in practice people really do want to pass callbacks into the native code, inherit from native code interfaces in Python, even something as simple as forwarding the logging in their native code back to Python logging (e.g. all of the really useful behavior possible with binding tools like pybind11). All of these are impossible to parallelize effectively today.
  - fulafel 3y ago
    The per-subinterpreter GIL that shipped in 3.12 should also help with situations where native code frameworks want to have parallel callbacks to Python code. But your logging use case is a good example of the problems because the Python code would still have to solve synchronisation there even without a GIL.
fulafel 3y ago
Threads + locks style heisenbug prone low-level programming is one possible avenue towards exploiting this, but its' notoriously difficult and a bad fit for Python's user base. Also, most parallelism is found on the GPU platforms. The alternative road is to figure out how Python could automatically exploit parallelism in the underlying hardware, possibly in a way that would let it work on GPUs as well. The SIMT data-parallel way (seen in eg in ISPC, OpenCL, shader languages) is also more programmer friendly as it doesn't require the constant use of error probe synchronisation primitives. Or other HLL approaches in Jax, Futhark, etc.
- Too 3y ago
  This is the goal of Mojo. Take existing Python code and compile it to the gpu.
  - machinekob 3y ago
    But they dont have GPU backend and most of the basic stuff isnt useful on GPU.
paulddraper 3y ago
> Python needs to have an urgent answer to the use of multiple cores https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html
- KaiserPro 3y ago
  Its not really an answer. if you're doing multiprocessing, you're better off having just having separate processes and use a message bus to do IPC. There are caveats, but multiprocessing rarely gives you enough extra for the overhead.
  - paulddraper 3y ago
    > you're better off having just having separate processes and use a message bus to do IPC Why are you better off that way?
    - KaiserPro 3y ago
      The sharing of data between processes in multi-processing, is done via posix pipe. This means that any quantity of data interchange incurs a heavy penalty. So the more you try and coordinate, the slower it gets. also `import logging` acts funny as well.
dragonwriter 3y ago
> In my opinion, Python needs to have an urgent answer to the use of multiple cores. Subinterpreters are an answer (heck, so is multiprocessing). Whether between them they are enough for Python's domains is another question. Probably not for the long term, but possibly for the neart term. But anything more is going to be a big lift, no-GIL is the obvious broader answer, so its good its being worked on, because by the time its betond question that its needed, it’ll be too late to start working in earnest.
- dalf 3y ago
  For now the suggested communication protocol is: * os.pipe and serialization (pickle or whatever): https://peps.python.org/pep-0554/#synchronize-using-an-os-pipe https://peps.python.org/pep-0554/#synchronize-using-an-os-pi... * immortal object, but I don't see a way to create immortal object from Python (only from C). https://engineering.fb.com/2023/08/15/developer-tools/immortal-objects-for-python-instagram-meta/ https://engineering.fb.com/2023/08/15/developer-tools/immort... I guess it will more iteration to get a better way to communicate between the interpreters.
tgv 3y ago
How about not using Python when you need performance? That 50% performance increase sounds nice, but it's still slow.
- KaiserPro 3y ago
  Thats basically what pybind is for.
  - sitkack 3y ago
    Take a look at https://github.com/wjakob/nanobind https://github.com/wjakob/nanobind > More concretely, benchmarks show up to ~4× faster compile time, ~5× smaller binaries, and ~10× lower runtime overheads compared to pybind11.

by bratao 3y ago

brianwawok 3y ago

My server can easily fill all 96 running standard Python under gunicorn.

NegativeLatency 3y ago
and if you use and async framework it's even more capable
iterateoften 3y ago
gunicorn uses multiple processes
- aeyes 3y ago
  You can still share most of the memory if you load everything before the fork so why is having multiple processes a problem?
  - rtpg 3y ago
    Ive found in practice that memory sharing doesn’t work as much as one would want, and multiple processes cost hundreds of megs (annoying!)
  - skrause 3y ago
    One operating system I have to deploy on (Windows) has neither fork() nor gunicorn.
  - dontlaugh 3y ago
    Reference counting prevents that from working well. Very little sharing actually happens by default.
    - sitkack 3y ago
      If the counts were kept somewhere else, they wouldn't pollute, or if you could turn off ref counts for everything allocated before the call to fork, then you would also be good. The other option is you could allocate all the shared data off of the Python heap prior to the fork call.
      - dontlaugh 3y ago
        Instagram have a system for making some objects immortal before the fork, but it’s not the default behaviour in Python. It’s also not easy to set up in some cases.
        aeyes 3y ago
        PEP683 finally landed in Python 3.12: https://peps.python.org/pep-0683/ https://peps.python.org/pep-0683/ It's been in Pyston for a while.
b5n 3y ago
Even simpler: parallel -j96 'python -c "print('{}')"' ::: $(for i in {1..96};do echo $i;done)

dekhn 3y ago

I've changed my opinion about this after many years. A long time ago IronPython was released which showed that you could build a high performance Python interpreter that was GIL-free. It had thread-safe containers (so multiple threads could work against the same lists, dictionaries, etc) and in some cases was faster than CPython (it was implemented using CLR and .net) When I saw IronPython I was immediately convinced that CPython should be the same way- already low-cost dual and quad core Intel machines were becoming available and it seemed like core counts were going to increase faster than clock rates. I figured that a small hit to serial performance would be more than acceptable if people could write multithreaded systems in Python, in much the same way as I wrote multithreaded C++. Over time after watching nogil not going anywhere in CPython (the python leadership didn't want to do nogil), with concomitant speedups in the single-processor implementation, along with the increasing use of C++ code that releases the GIL, and seeing that many people just weren't good at multithreaded programming, I have started to conclude that the multithreading/multiprocess in Python today is about the best we can get. That is, instead of having threaded containers and multiple interpreter threads all banging on the same underlying data, it's a lot easier to just use threads as work queues that have minimal interaction with other threads. So that's where I've ended up: some of my code using multiprocessing, typically the Pool or ThreadPool, with the concurrent future API to handle result-gathering, barriers, etc. Other code has an external system that starts many python processes from the command line and waits for those processes to complete. Other code is single-threaded in python and launches C++ cores that launch multiple threads just to return a computational result to python faster. And I think trying to do both gil and nogil interpreters, rather than committing to one or the other, the python leadership will sign us up for untold inconveniences around packaging. We already see this in the move to async and it will only be worse with threading. So sad to say I think sticking to GIL and using the approaches I mentioned above (along with others that work well for concurrent, rather than parallel, computing, like coroutines) is the best thing to do right now and I'm a bit bummed that the leadership signalled their intent to accomodate both.

alfalfasprout 3y ago
You've been able to release GIL for a long time though in the C ABI (and C++ via pybind) and that hasn't removed the need for nogil. The reason is that a single thread can invoke multicore c/c++ code (even calling into specialized accelerators if needed) but having python objects that are shared between threads is extremely clunky. And multiprocessing results in a lot of communication overhead. Worse-- python code cannot proceed in another thread. Mixing python and C++ is very common in ML and scientific computing workloads. > And I think trying to do both gil and nogil interpreters, rather than committing to one or the other, the python leadership will sign us up for untold inconveniences around packaging. We already see this in the move to async and it will only be worse with threading. How has packaging been affected by async? As long as you have a compatible python version what issue do you run into?

jshen 3y ago

Why? What are the use cases? I honestly can't think of any. If you are making a web app, you can use those cores just fine. If you are doing ML you will be calling into native code which can also do it just fine. If you are trying to make a AAA game, you shouldn't use python, etc. Not sure what the use case is.

cpgxiii 3y ago
> If you are doing ML you will be calling into native code which can also do it just fine. This is exactly the use case. You can only parallelize in the native code if the boundary between Python and native code is absolute. But in practice people really do want to pass callbacks into the native code, inherit from native code interfaces in Python, even something as simple as forwarding the logging in their native code back to Python logging (e.g. all of the really useful behavior possible with binding tools like pybind11). All of these are impossible to parallelize effectively today.
- fulafel 3y ago
  The per-subinterpreter GIL that shipped in 3.12 should also help with situations where native code frameworks want to have parallel callbacks to Python code. But your logging use case is a good example of the problems because the Python code would still have to solve synchronisation there even without a GIL.

fulafel 3y ago

Threads + locks style heisenbug prone low-level programming is one possible avenue towards exploiting this, but its' notoriously difficult and a bad fit for Python's user base. Also, most parallelism is found on the GPU platforms. The alternative road is to figure out how Python could automatically exploit parallelism in the underlying hardware, possibly in a way that would let it work on GPUs as well. The SIMT data-parallel way (seen in eg in ISPC, OpenCL, shader languages) is also more programmer friendly as it doesn't require the constant use of error probe synchronisation primitives. Or other HLL approaches in Jax, Futhark, etc.

Too 3y ago
This is the goal of Mojo. Take existing Python code and compile it to the gpu.
- machinekob 3y ago
  But they dont have GPU backend and most of the basic stuff isnt useful on GPU.

paulddraper 3y ago

> Python needs to have an urgent answer to the use of multiple cores https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html

KaiserPro 3y ago
Its not really an answer. if you're doing multiprocessing, you're better off having just having separate processes and use a message bus to do IPC. There are caveats, but multiprocessing rarely gives you enough extra for the overhead.
- paulddraper 3y ago
  > you're better off having just having separate processes and use a message bus to do IPC Why are you better off that way?
  - KaiserPro 3y ago
    The sharing of data between processes in multi-processing, is done via posix pipe. This means that any quantity of data interchange incurs a heavy penalty. So the more you try and coordinate, the slower it gets. also `import logging` acts funny as well.

dragonwriter 3y ago

> In my opinion, Python needs to have an urgent answer to the use of multiple cores. Subinterpreters are an answer (heck, so is multiprocessing). Whether between them they are enough for Python's domains is another question. Probably not for the long term, but possibly for the neart term. But anything more is going to be a big lift, no-GIL is the obvious broader answer, so its good its being worked on, because by the time its betond question that its needed, it’ll be too late to start working in earnest.

dalf 3y ago
For now the suggested communication protocol is: * os.pipe and serialization (pickle or whatever): https://peps.python.org/pep-0554/#synchronize-using-an-os-pipe https://peps.python.org/pep-0554/#synchronize-using-an-os-pi... * immortal object, but I don't see a way to create immortal object from Python (only from C). https://engineering.fb.com/2023/08/15/developer-tools/immortal-objects-for-python-instagram-meta/ https://engineering.fb.com/2023/08/15/developer-tools/immort... I guess it will more iteration to get a better way to communicate between the interpreters.

tgv 3y ago

How about not using Python when you need performance? That 50% performance increase sounds nice, but it's still slow.

KaiserPro 3y ago
Thats basically what pybind is for.
- sitkack 3y ago
  Take a look at https://github.com/wjakob/nanobind https://github.com/wjakob/nanobind > More concretely, benchmarks show up to ~4× faster compile time, ~5× smaller binaries, and ~10× lower runtime overheads compared to pybind11.