3 ms·

> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system)

> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle. You're conflating several things that are orthogonal imo. A system can be distributed without concurrency. A concurrent system need not be distributed. Either kind of thing can be built with or without a specific kind of type system. And CPU efficiency has nothing to do specifically with any of the previous things. To expand on that: distributed systems are quite often constructed from simple single-threaded processes, and where concurrency is needed it is probably more often achieved through multi-processing than multi-threading. A single-threaded event-dispatched request-response service probably describes a big chunk of all the stuff running in distributed environments today. In a lot of these cases the workloads are i/o bound, and instructions per cycle is not even close to the top of the list of concerns. There are a lot of reasons why python fits into that world very well.

badpun 7y ago
> A system can be distributed without concurrency. A concurrent system need not be distributed. The only justification for such design I can think of is HA. I.e. you have very light workload that does not saturate a single machine, but nonetheless you create a distributed system with two machines for HA. Such light workload is probably not often the case at Netflix.
- markbnj 7y ago
  > The only justification for such design I can think of is HA. I'm not trying to justify any particular design. I'm just saying that concurrency and distributed processing are different things. And I'm no expert, but I continue to be confused as to what saturation, whether of cpu or i/o - you don't specify - has to do with concurrency? If I have for example n http servers all running one thread and handling enough traffic that they are all at 100 percent CPU that is by some minimal definition a distributed system, but none of the processes in it are concurrent. If the http servers read and write a database then the database is almost certainly concurrent, so then you have a distributed system that has both concurrent and non-concurrent processes collaborating.
  - badpun 7y ago
    > f I have for example n http servers all running one thread and handling enough traffic that they are all at 100 percent CPU that is by some minimal definition a distributed system, but none of the processes in it are concurrent. Concurrency is broader than just threads in a process. In your example, the whole problem is concurrent.
    - markbnj 7y ago
      Yes of course you're correct. Any system of that kind is concurrent across its architecture. But the current context being considered is that of concurrency as a programming paradigm, which was established by the OP's observations on python's lack of concurrency which sparked the exchange. In the comment you quoted above the statement "none of the processes are concurrent" was meant to make the context clearer.

badpun 7y ago

> A system can be distributed without concurrency. A concurrent system need not be distributed. The only justification for such design I can think of is HA. I.e. you have very light workload that does not saturate a single machine, but nonetheless you create a distributed system with two machines for HA. Such light workload is probably not often the case at Netflix.

markbnj 7y ago
> The only justification for such design I can think of is HA. I'm not trying to justify any particular design. I'm just saying that concurrency and distributed processing are different things. And I'm no expert, but I continue to be confused as to what saturation, whether of cpu or i/o - you don't specify - has to do with concurrency? If I have for example n http servers all running one thread and handling enough traffic that they are all at 100 percent CPU that is by some minimal definition a distributed system, but none of the processes in it are concurrent. If the http servers read and write a database then the database is almost certainly concurrent, so then you have a distributed system that has both concurrent and non-concurrent processes collaborating.
- badpun 7y ago
  > f I have for example n http servers all running one thread and handling enough traffic that they are all at 100 percent CPU that is by some minimal definition a distributed system, but none of the processes in it are concurrent. Concurrency is broader than just threads in a process. In your example, the whole problem is concurrent.
  - markbnj 7y ago
    Yes of course you're correct. Any system of that kind is concurrent across its architecture. But the current context being considered is that of concurrency as a programming paradigm, which was established by the OP's observations on python's lack of concurrency which sparked the exchange. In the comment you quoted above the statement "none of the processes are concurrent" was meant to make the context clearer.