5 ms·

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other

If Python is slow and you need concurrency, and you have a long-ish timeframe in mind, do yourself a favor and use one of C++/Java/Rust/Go/Erlang/Elixir/(other newer programming language). Especially, don’t rely on Multiprocessing. I’ve been developing and maintaining a largish batch processing Python project where performance is a key feature, and it’s been very frustrating. I should have rewritten it when I had the chance, but I’m slowly outsourcing critical pieces to C++ and Rust libraries. Multiprocessing has been a source of subtle portability errors as it has changed over different minor versions.

wenc 4y ago
If you need to stick with Python, Joblib is more reliable and works better than Multiprocessing in my experience. I ran into so many subtle quirks that needed workarounds in Multiprocessing. Joblib.parallel with the Loky backend just seems to be Multiprocessing done right.
jeremycarter 4y ago
I have a similar project and now have to split up the single application to actually get the muli core processing I demand. I regret not using Rust/Go... or even just .NET.
ansre 4y ago
[dead]
- jonstewart 4y ago
  Scaling horizontally is well and good (since my unit of input is large and I get a lot of them, I just use autoscaling groups in AWS to achieve much the same), but I don’t want to pay for 10 systems slowly doing what 1 could do with fast software. Especially, while 1 system has a hard ceiling, there’s a lot of headroom on 1 system when you achieve mechanical sympathy with the cache hierarchy, memory controller, and disk.
capableweb 4y ago
Hear hear! If you already know Python, picking up the basics of Java/Rust/Go/Elixir is trivial. Picking up the basics of C++ would be slightly harder, but not impossible. Erlang even further away, at least syntax wise. But, you'll be doing future you a huge service for whenever you need to reach for hard performance/latency and/or concurrency/parallelism again.
AceJohnny2 4y ago
I think I need Python concurrency because I'm using it to talk to a bunch of hardware in parallel (using pexpect, for testing), collecting global statistics. I'm not using concurrency for performance (well, not in the usual way I guess). Using a different language would put me at odds with the rest of the team, increasing the maintenance burden. What solution would people recommend?
- henryfjordan 4y ago
  Write up the same simple example in Python and in a different language that is better suited to the task. If the other language is that much easier, it should be easy to convince people with the example. Often times you think "since my team is using Python that'll be easiest" but if you have to rely on libraries (even parts of the standard lib) that your teammates don't often use then you might not be getting as much benefit as you'd think. A practical example might show everyone that switching languages to get access to a better interface for your problem might be easier for everyone. It might also show you that Python is actually the right choice, so it's a good exercise.
- jonstewart 4y ago
  That seems like a better application for Python concurrency than trying to eke out more performance. I don’t particularly like Go, but it might be worth a look for this since it’s somewhat easier to learn and may be more accessible for others to learn and maintain. I wouldn’t do a full implementation, just an exploratory project for the sake of seeing whether you’re stuck on a local maxima with Python.
vajenetehais 4y ago
Genuinely asking, why don't you add dotnet ? I'm currently in the situation you describe and I would like to try something new. Dotnet is among my candidates. (only asking for personal projects)
- jonstewart 4y ago
  Mostly because I forgot it.
deleted 4y ago
[deleted]

wenc 4y ago

If you need to stick with Python, Joblib is more reliable and works better than Multiprocessing in my experience. I ran into so many subtle quirks that needed workarounds in Multiprocessing. Joblib.parallel with the Loky backend just seems to be Multiprocessing done right.

jeremycarter 4y ago

I have a similar project and now have to split up the single application to actually get the muli core processing I demand. I regret not using Rust/Go... or even just .NET.

ansre 4y ago

[dead]

jonstewart 4y ago
Scaling horizontally is well and good (since my unit of input is large and I get a lot of them, I just use autoscaling groups in AWS to achieve much the same), but I don’t want to pay for 10 systems slowly doing what 1 could do with fast software. Especially, while 1 system has a hard ceiling, there’s a lot of headroom on 1 system when you achieve mechanical sympathy with the cache hierarchy, memory controller, and disk.

capableweb 4y ago

Hear hear! If you already know Python, picking up the basics of Java/Rust/Go/Elixir is trivial. Picking up the basics of C++ would be slightly harder, but not impossible. Erlang even further away, at least syntax wise. But, you'll be doing future you a huge service for whenever you need to reach for hard performance/latency and/or concurrency/parallelism again.

AceJohnny2 4y ago

I think I need Python concurrency because I'm using it to talk to a bunch of hardware in parallel (using pexpect, for testing), collecting global statistics. I'm not using concurrency for performance (well, not in the usual way I guess). Using a different language would put me at odds with the rest of the team, increasing the maintenance burden. What solution would people recommend?

henryfjordan 4y ago
Write up the same simple example in Python and in a different language that is better suited to the task. If the other language is that much easier, it should be easy to convince people with the example. Often times you think "since my team is using Python that'll be easiest" but if you have to rely on libraries (even parts of the standard lib) that your teammates don't often use then you might not be getting as much benefit as you'd think. A practical example might show everyone that switching languages to get access to a better interface for your problem might be easier for everyone. It might also show you that Python is actually the right choice, so it's a good exercise.
jonstewart 4y ago
That seems like a better application for Python concurrency than trying to eke out more performance. I don’t particularly like Go, but it might be worth a look for this since it’s somewhat easier to learn and may be more accessible for others to learn and maintain. I wouldn’t do a full implementation, just an exploratory project for the sake of seeing whether you’re stuck on a local maxima with Python.

vajenetehais 4y ago

Genuinely asking, why don't you add dotnet ? I'm currently in the situation you describe and I would like to try something new. Dotnet is among my candidates. (only asking for personal projects)

jonstewart 4y ago
Mostly because I forgot it.

deleted 4y ago

[deleted]