5 ms·

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded perfor

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

oofbey 4y ago
On the flip side, if your workload can be parallelized across thousands of cores, python has about the best CUDA support anywhere.
- pjmlp 4y ago
  That would be C++ and Fortran actually.
  - p10_user 4y ago
    But the python bindings are great and useful to many, so Python gets to be added to the list.
    - pjmlp 4y ago
      Just like any language with FFI capabilities to call the same libraries.
Nimitz14 4y ago
As another python dev, it has basically never been a painpoint for me. Your anecdote adds little.
IceWreck 4y ago
Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.
- xwdv 4y ago
  I would have thought convincing people they’ll just have to use Go or Rust or Elixir would have been an easy sell around here. Turns out they just want a better Python.
  - fnord123 4y ago
    >Turns out they just want a better Python. That's Go. It gives actual types[1] and structs (so you dont have to wonder about dict, class, class with slots, dataclasses, pydantic, attrs, cattrs, marshmallow, etc). It removes exceptions and monkeypatching. It's async-first (a bit like gevent). It's inherently multicore. And you can package and distribute it without marking a pentagram on the ground and sacrificing an intern to Apep. You just need to stop being friends with C FFI. Which is fine for web and sysadmin tools. For data science and ML/AI/DL, it's less ok. [1] And the types are actually checked! I think a lot of people aren't really using python type checking given how slow it is and no one seems to complain. Or maybe everyone is using pyright and pyre and are satisfied with how slow these are.
    - kaba0 4y ago
      Going to the very unexpressive Go from the expressivity of python is a goddamn huge jump though. Going to JS or even TS for performance would be saner, and it has a same-ish object model even.
      - fnord123 4y ago
        The expressivity in Python is a problem that needs to be solved though. Moving to JS goes the wrong way.
  - pjmlp 4y ago
    It already exists, but they don't want to learn other languages.
- Mehdi2277 4y ago
  Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly. It's also commonly used language for numerical work in general. Most of the time numpy is enough and then occasionally you'll need something not already implemented and then have to do your own bindings.
Galanwe 4y ago
> As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Hmm, how is that so? As a python dev as well, I don't have much complaint with multiprocessing. The API is simple, it works OK, the overall paradigm is simple to grok, you can share transparently with pickle, etc.
- nine_k 4y ago
  Multiprocessing is fine, but the cost of starting another interpreter is pretty visible, so you need to create a pool, and it may not be an overall speedup if the run time is short. It takes more careful planning than async in JS?, say, or goroutines.
  - geysersam 4y ago
    Yeah but for JS style async you'd use probably use an event loop in Python, not multi processing.
    - nine_k 4y ago
      Yes. But, frankly, async is also simpler in JS than in Python: e.g. no need to start a reactor loop.
      - Too 4y ago
        Starting the event loop is no worse than any setup of a main function, it’s a oneliner: asyncio.get_event_loop().run_until_completion(my_async_main)
        BerislavLopac 4y ago
        Errr no, that has been replaced with asyncio.run quite some time ago.
- plonk 4y ago
  Pickle isn't transparent though, custom objects that wrap files or database sessions need to override serialization. The ProcessPoolExecutor is nice but shouldn't be necessary.

oofbey 4y ago

On the flip side, if your workload can be parallelized across thousands of cores, python has about the best CUDA support anywhere.

pjmlp 4y ago
That would be C++ and Fortran actually.
- p10_user 4y ago
  But the python bindings are great and useful to many, so Python gets to be added to the list.
  - pjmlp 4y ago
    Just like any language with FFI capabilities to call the same libraries.

Nimitz14 4y ago

As another python dev, it has basically never been a painpoint for me. Your anecdote adds little.

IceWreck 4y ago

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

xwdv 4y ago
I would have thought convincing people they’ll just have to use Go or Rust or Elixir would have been an easy sell around here. Turns out they just want a better Python.
- fnord123 4y ago
  >Turns out they just want a better Python. That's Go. It gives actual types[1] and structs (so you dont have to wonder about dict, class, class with slots, dataclasses, pydantic, attrs, cattrs, marshmallow, etc). It removes exceptions and monkeypatching. It's async-first (a bit like gevent). It's inherently multicore. And you can package and distribute it without marking a pentagram on the ground and sacrificing an intern to Apep. You just need to stop being friends with C FFI. Which is fine for web and sysadmin tools. For data science and ML/AI/DL, it's less ok. [1] And the types are actually checked! I think a lot of people aren't really using python type checking given how slow it is and no one seems to complain. Or maybe everyone is using pyright and pyre and are satisfied with how slow these are.
  - kaba0 4y ago
    Going to the very unexpressive Go from the expressivity of python is a goddamn huge jump though. Going to JS or even TS for performance would be saner, and it has a same-ish object model even.
    - fnord123 4y ago
      The expressivity in Python is a problem that needs to be solved though. Moving to JS goes the wrong way.
- pjmlp 4y ago
  It already exists, but they don't want to learn other languages.
Mehdi2277 4y ago
Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly. It's also commonly used language for numerical work in general. Most of the time numpy is enough and then occasionally you'll need something not already implemented and then have to do your own bindings.

Galanwe 4y ago

> As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Hmm, how is that so? As a python dev as well, I don't have much complaint with multiprocessing. The API is simple, it works OK, the overall paradigm is simple to grok, you can share transparently with pickle, etc.

nine_k 4y ago
Multiprocessing is fine, but the cost of starting another interpreter is pretty visible, so you need to create a pool, and it may not be an overall speedup if the run time is short. It takes more careful planning than async in JS?, say, or goroutines.
- geysersam 4y ago
  Yeah but for JS style async you'd use probably use an event loop in Python, not multi processing.
  - nine_k 4y ago
    Yes. But, frankly, async is also simpler in JS than in Python: e.g. no need to start a reactor loop.
    - Too 4y ago
      Starting the event loop is no worse than any setup of a main function, it’s a oneliner: asyncio.get_event_loop().run_until_completion(my_async_main)
      - BerislavLopac 4y ago
        Errr no, that has been replaced with asyncio.run quite some time ago.
plonk 4y ago
Pickle isn't transparent though, custom objects that wrap files or database sessions need to override serialization. The ProcessPoolExecutor is nice but shouldn't be necessary.