5 ms·

I recently tried to parallelize some non trivial code I have with multiprocessing.map/imap/imap_unordered - the state of parallel computing in python is really

I recently tried to parallelize some non trivial code I have with multiprocessing.map/imap/imap_unordered - the state of parallel computing in python is really poor have to say. If the program's CPU runtime has gotten so long that you'd think about parallelizing, its datastructures tend to be too much to push through the pickle bottleneck and no gain would be had. Except of course number crunching stuff for which there's numpy. Bottom line is, you can do parallel computing, but trying to parallelize after the fact will usually fail, you have to plan for it from the very beginning due to no shared memory, which would add significant time until your first version is up and running I long for a dynamic but strictly typed language with rich standard lib that gets this right. Nim?

bmh100 10y ago
You should consider Clojure with its optional typing library. I used Clojure, and later converted a single-threaded application into multi-threaded with dramatic gains. I did not need to plan in advance for parallelization and did not do so. Despite that, the conversion was fairly pain free. The application did the following: 1. read csv 2. create data structure 3. calculate statistics for each column (max values, unique values, etc.) 4. combine those statistics with original columns to generate new columns 5. write new columns to new csv Processing each type of statistics in its own thread, combined with promises, allowed the processors to independently work on the same data structure. Because of the need to handle non-numeric values in numeric columns, e.g. convert nulls to zeroes, matrix manipulations would not have been appropriate. I just wanted to point out a type of situation from my experience that goes against your generalization, to give a more complete picture.
- m_mueller 10y ago
  Thanks, I'll consider clojure in the future. How would you say its standard lib and third party libraries hold up against python? With my point about the need to plan ahead I specifically meant python by the way, and I think all runtimes that can't make use of concurrent multithreading have this in common. As soon as you can do multithreading with shared memory you become a lot more flexible in parallelising later on, and you can adjust your code style accordingly to accommodate this without much overhead. That's why the GIL IMO is so unfortunate, it's a real shame that Python 3 wasn't able to get rid of this when it started with a clean slate.
  - bmh100 10y ago
    I see now what you meant. I apologize for my misunderstanding in that to be a general statement. Knowing that, I agree with your statements much more now. From what I can tell, the multi-process architecture in Python would require everything you mentioned. Not removing the GIL is indeed unfortunate. Clojure is niche in a way. It is a Lisp dialect. However, it is based on the JVM and has built-in conveniences for using Java functions, classes, and libraries. Therefore, comparing libraries for enterprise applications is more accurately Clojure+Java vs. Python. My expertise is not in either language ecosystem, but my impression is that Python has better libraries for data analysis and as a system scripting language, and Clojure+Java is better for general applications and everything else. One benefit of Clojure that makes parallelization easy is the immutable data structures. New data structures are cheaply made by seamlessly referencing originating data structures. Example: making a copy of array A and appending an item as array B, means that array B is mostly a reference to array A. Immutable data structures also offer expectations of behavior that mutable state naturally offends, such as avoiding race conditions or locks. In the case of my earlier example, I simply added some "future" wrapper functions and "dereference" annotations [1]. The entire program would execute to reach the final output, then work in reverse order to trigger and plan the future functions. Order of execution was based on dependencies, not strictly on the order of the code. It was fascinating was to watch the logs for when functions would start or complete. If you are familiar with operations planning and critical path analysis, the overall run-time reflected the critical path. [1]: https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/clojure.core/future https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/c...
nimmer 10y ago
Although the standard lib is still improving, Nim is doing it right.
srean 10y ago
Its a bit immature at the moment, but here's a shout out for Julia. If you are used to python, numpy you will not feel out of place. BTW, as far as I recall Nim is statically typed, please correct me if I am wrong.
- m_mueller 10y ago
  Yes, Nim is static, however I've been told using lots of type inference and unions it can get almost the ease of use of dynamic languages. I never did something with it so far though, so I can't say whether that really holds up. My main concern would still be libraries - hard to beat python or the JVM in that regard I think.

bmh100 10y ago

You should consider Clojure with its optional typing library. I used Clojure, and later converted a single-threaded application into multi-threaded with dramatic gains. I did not need to plan in advance for parallelization and did not do so. Despite that, the conversion was fairly pain free. The application did the following: 1. read csv 2. create data structure 3. calculate statistics for each column (max values, unique values, etc.) 4. combine those statistics with original columns to generate new columns 5. write new columns to new csv Processing each type of statistics in its own thread, combined with promises, allowed the processors to independently work on the same data structure. Because of the need to handle non-numeric values in numeric columns, e.g. convert nulls to zeroes, matrix manipulations would not have been appropriate. I just wanted to point out a type of situation from my experience that goes against your generalization, to give a more complete picture.

m_mueller 10y ago
Thanks, I'll consider clojure in the future. How would you say its standard lib and third party libraries hold up against python? With my point about the need to plan ahead I specifically meant python by the way, and I think all runtimes that can't make use of concurrent multithreading have this in common. As soon as you can do multithreading with shared memory you become a lot more flexible in parallelising later on, and you can adjust your code style accordingly to accommodate this without much overhead. That's why the GIL IMO is so unfortunate, it's a real shame that Python 3 wasn't able to get rid of this when it started with a clean slate.
- bmh100 10y ago
  I see now what you meant. I apologize for my misunderstanding in that to be a general statement. Knowing that, I agree with your statements much more now. From what I can tell, the multi-process architecture in Python would require everything you mentioned. Not removing the GIL is indeed unfortunate. Clojure is niche in a way. It is a Lisp dialect. However, it is based on the JVM and has built-in conveniences for using Java functions, classes, and libraries. Therefore, comparing libraries for enterprise applications is more accurately Clojure+Java vs. Python. My expertise is not in either language ecosystem, but my impression is that Python has better libraries for data analysis and as a system scripting language, and Clojure+Java is better for general applications and everything else. One benefit of Clojure that makes parallelization easy is the immutable data structures. New data structures are cheaply made by seamlessly referencing originating data structures. Example: making a copy of array A and appending an item as array B, means that array B is mostly a reference to array A. Immutable data structures also offer expectations of behavior that mutable state naturally offends, such as avoiding race conditions or locks. In the case of my earlier example, I simply added some "future" wrapper functions and "dereference" annotations [1]. The entire program would execute to reach the final output, then work in reverse order to trigger and plan the future functions. Order of execution was based on dependencies, not strictly on the order of the code. It was fascinating was to watch the logs for when functions would start or complete. If you are familiar with operations planning and critical path analysis, the overall run-time reflected the critical path. [1]: https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/clojure.core/future https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/c...

nimmer 10y ago

Although the standard lib is still improving, Nim is doing it right.

srean 10y ago

Its a bit immature at the moment, but here's a shout out for Julia. If you are used to python, numpy you will not feel out of place. BTW, as far as I recall Nim is statically typed, please correct me if I am wrong.

m_mueller 10y ago
Yes, Nim is static, however I've been told using lots of type inference and unions it can get almost the ease of use of dynamic languages. I never did something with it so far though, so I can't say whether that really holds up. My main concern would still be libraries - hard to beat python or the JVM in that regard I think.