4 ms·

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a gr

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

KZerda 5y ago
Lots of reasons. Sometimes concurrency is a relatively soft need where you could get by with multiple processes, but it would be nice if the language itself provided some capability for things like threads. Or your dev team is much more familiar with python than with other languages, and the time to retrain and rewrite would be greater than the benefits. Or the rest of the language could have greater issues and you don't want to give up excellent libraries like numpy.
ska 5y ago
To a first approximation, people don't use python for itself, they use it for the vast ecosystem and network effect. If you jump to another language for better concurrency, what are you giving up? Unless you really are doing greenfield development in an isolated application, these considerations often trump any language feature.
- otterley 5y ago
  Don't get me wrong; I'm not suggesting that anyone dump Python altogether to switch to a different language for any arbitrary project or purpose. Many businesses I work with use different languages for different components or applications, using the network or storage (or even shared memory) to intercommunicate when necessary. The right tool for the job, as it were.
  - deleted 5y ago
    [deleted]
  - Mehdi2277 5y ago
    I use python mostly for numpy/tensorflow/etc. The machine learning ecosystem. That ecosystem cares a lot about multithreaded performance. So historically answer has been write c/c++ and then bind to python. This work is mainly motivated by libraries like that wanting to write less c extensions and be able to just write python and still have proper thread performance.
    - otterley 5y ago
      As I mention elsewhere, I think what they care about is parallelism, not concurrency, and you don’t need threads to attain that efficiently.
ferdowsi 5y ago
There are some organizations with lots of domain knowledge and expertise around around developing, securing and deploying Python and they don't have the Innovation Currency to spend on investing in a new language. Specific to your point, recruiting for Elixir talent is a problem compared to more mainstream languages. Recruiting in general is extremely hard at this moment.
- otterley 5y ago
  Given all the corner cases people are going to continue to find whilst trying to coax Python into behaving correctly in a highly concurrent program -- especially one that utilizes random libraries from the ecosystem -- I can't help but wonder whether the Innovation Currency is better spent replacing the components that require high concurrency (which often is only a subset of them) instead of getting stuck in the mire of bug-smashing.
pmontra 5y ago
A possible answer is that everybody in the company knows Python and no other language. Another one is that they have to reuse or extend a bunch of existing Python code. The latter happened to me. Performances were definitely not a concern but I suddenly needed threads doing extra functionality over the original single threaded algorithm. BTW, I used a queue to pass messages between them.
- otterley 5y ago
  Using multiple interpreters with message passing is a workable, if expensive, way to deal with the problem. It is trading one cost for another. (These sort of tradeoffs are encountered all the time in business, to be sure.)
Fordec 5y ago
Sometimes a project starts off aiming to solve a problem. maybe it's a data science problem, so support already exists in python so lets do that. Ok, it worked great and it's catching on with users. Now we need to scale, but we are running into concurrency issues. What is a better answer? Ok we will work on improving python concurrency under the hood, or completely scrap the code base and switch to a different language? Very few people set out going asking themselves about such low level details on day one of a project. Especially something that was an MVP or POC
- otterley 5y ago
  I'll plead ignorance here: Do data science workflows often require high concurrency using a single interpreter? I thought all that stuff was compute-bound and parceled out to workers that farm out calculations to CPUs and GPUs.
  - Mehdi2277 5y ago
    Yes they do. I’ve written cython/numba as work arounds before. A lot of times if you need a small operation done many times the multiprocessor overhead is bad, but writing a pure python for loop over numpy/other tensors is awful for performance. The answer historically has been c/c++ and bind to python. This work is mainly motivated by one of those libraries wanting to write less c++ bindings and be able to do operations like these parallel directly in python.
    - otterley 5y ago
      This doesn’t sound right. I think you’re mixing up parallelism/multi-core/linear resource scalability with concurrency. Usually you want high concurrency to handle multiplexing events where a thread or process per client would be waiting idle the majority of the time. ML is compute bound so by definition there wouldn’t be idle threads. And you can already get multicore work done in Python by simply using its venerable multiprocessing library. Threading or cooperative scheduling doesn’t really buy you much in this case. And in fact the NumPy documentation confirms this.
klyrs 5y ago
I rarely need concurrency, and do a lot of Python because it's what all my dependencies are written in. But sometimes, I find myself bottlenecked on a trivially parallelizable operation. In the state (my dependecies are in Python, I have a working Python implementation), there's no way in hell that (rewrite my dependencies in Elixir, rewrite my code in Elixir) is a sensible next move.
lucb1e 5y ago
Are you proposing to write anything that will need concurrency anywhere in your favorite language, or just call into the concurrent code from python? (Since comments like https://news.ycombinator.com/item?id=28883990 https://news.ycombinator.com/item?id=28883990 seem to be taking it as the former whereas I took it as the latter.)
- otterley 5y ago
  Calling in from Python is a totally reasonable option! See my comment here: https://news.ycombinator.com/item?id=28883591 https://news.ycombinator.com/item?id=28883591
dragonwriter 5y ago
> This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Because sometimes what you really need, or at least want, is an overlap of concurrency and other things, and Python is optimal for you for the other things. If you just need concurrency, sure, but how often is that the only requirement?
phyrex 5y ago
Because we started with python for historical reasons when the company was small and performance didn’t matter, and now we’re stuck with it

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

KZerda 5y ago

Lots of reasons. Sometimes concurrency is a relatively soft need where you could get by with multiple processes, but it would be nice if the language itself provided some capability for things like threads. Or your dev team is much more familiar with python than with other languages, and the time to retrain and rewrite would be greater than the benefits. Or the rest of the language could have greater issues and you don't want to give up excellent libraries like numpy.

ska 5y ago

To a first approximation, people don't use python for itself, they use it for the vast ecosystem and network effect. If you jump to another language for better concurrency, what are you giving up? Unless you really are doing greenfield development in an isolated application, these considerations often trump any language feature.

otterley 5y ago
Don't get me wrong; I'm not suggesting that anyone dump Python altogether to switch to a different language for any arbitrary project or purpose. Many businesses I work with use different languages for different components or applications, using the network or storage (or even shared memory) to intercommunicate when necessary. The right tool for the job, as it were.
- deleted 5y ago
  [deleted]
- Mehdi2277 5y ago
  I use python mostly for numpy/tensorflow/etc. The machine learning ecosystem. That ecosystem cares a lot about multithreaded performance. So historically answer has been write c/c++ and then bind to python. This work is mainly motivated by libraries like that wanting to write less c extensions and be able to just write python and still have proper thread performance.
  - otterley 5y ago
    As I mention elsewhere, I think what they care about is parallelism, not concurrency, and you don’t need threads to attain that efficiently.

ferdowsi 5y ago

There are some organizations with lots of domain knowledge and expertise around around developing, securing and deploying Python and they don't have the Innovation Currency to spend on investing in a new language. Specific to your point, recruiting for Elixir talent is a problem compared to more mainstream languages. Recruiting in general is extremely hard at this moment.

otterley 5y ago
Given all the corner cases people are going to continue to find whilst trying to coax Python into behaving correctly in a highly concurrent program -- especially one that utilizes random libraries from the ecosystem -- I can't help but wonder whether the Innovation Currency is better spent replacing the components that require high concurrency (which often is only a subset of them) instead of getting stuck in the mire of bug-smashing.

pmontra 5y ago

A possible answer is that everybody in the company knows Python and no other language. Another one is that they have to reuse or extend a bunch of existing Python code. The latter happened to me. Performances were definitely not a concern but I suddenly needed threads doing extra functionality over the original single threaded algorithm. BTW, I used a queue to pass messages between them.

otterley 5y ago
Using multiple interpreters with message passing is a workable, if expensive, way to deal with the problem. It is trading one cost for another. (These sort of tradeoffs are encountered all the time in business, to be sure.)

Fordec 5y ago

Sometimes a project starts off aiming to solve a problem. maybe it's a data science problem, so support already exists in python so lets do that. Ok, it worked great and it's catching on with users. Now we need to scale, but we are running into concurrency issues. What is a better answer? Ok we will work on improving python concurrency under the hood, or completely scrap the code base and switch to a different language? Very few people set out going asking themselves about such low level details on day one of a project. Especially something that was an MVP or POC

otterley 5y ago
I'll plead ignorance here: Do data science workflows often require high concurrency using a single interpreter? I thought all that stuff was compute-bound and parceled out to workers that farm out calculations to CPUs and GPUs.
- Mehdi2277 5y ago
  Yes they do. I’ve written cython/numba as work arounds before. A lot of times if you need a small operation done many times the multiprocessor overhead is bad, but writing a pure python for loop over numpy/other tensors is awful for performance. The answer historically has been c/c++ and bind to python. This work is mainly motivated by one of those libraries wanting to write less c++ bindings and be able to do operations like these parallel directly in python.
  - otterley 5y ago
    This doesn’t sound right. I think you’re mixing up parallelism/multi-core/linear resource scalability with concurrency. Usually you want high concurrency to handle multiplexing events where a thread or process per client would be waiting idle the majority of the time. ML is compute bound so by definition there wouldn’t be idle threads. And you can already get multicore work done in Python by simply using its venerable multiprocessing library. Threading or cooperative scheduling doesn’t really buy you much in this case. And in fact the NumPy documentation confirms this.

klyrs 5y ago

I rarely need concurrency, and do a lot of Python because it's what all my dependencies are written in. But sometimes, I find myself bottlenecked on a trivially parallelizable operation. In the state (my dependecies are in Python, I have a working Python implementation), there's no way in hell that (rewrite my dependencies in Elixir, rewrite my code in Elixir) is a sensible next move.

lucb1e 5y ago

Are you proposing to write anything that will need concurrency anywhere in your favorite language, or just call into the concurrent code from python? (Since comments like https://news.ycombinator.com/item?id=28883990 https://news.ycombinator.com/item?id=28883990 seem to be taking it as the former whereas I took it as the latter.)

otterley 5y ago
Calling in from Python is a totally reasonable option! See my comment here: https://news.ycombinator.com/item?id=28883591 https://news.ycombinator.com/item?id=28883591

dragonwriter 5y ago

> This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Because sometimes what you really need, or at least want, is an overlap of concurrency and other things, and Python is optimal for you for the other things. If you just need concurrency, sure, but how often is that the only requirement?

phyrex 5y ago

Because we started with python for historical reasons when the company was small and performance didn’t matter, and now we’re stuck with it