3 ms·
You should consider Clojure with its optional typing library. I used Clojure, and later converted a single-threaded application into multi-threaded with dramati
by bmh100 10y ago
You should consider Clojure with its optional typing library. I used Clojure, and later converted a single-threaded application into multi-threaded with dramatic gains. I did not need to plan in advance for parallelization and did not do so. Despite that, the conversion was fairly pain free. The application did the following:
1. read csv
2. create data structure
3. calculate statistics for each column (max values, unique values, etc.)
4. combine those statistics with original columns to generate new columns
5. write new columns to new csv
Processing each type of statistics in its own thread, combined with promises, allowed the processors to independently work on the same data structure. Because of the need to handle non-numeric values in numeric columns, e.g. convert nulls to zeroes, matrix manipulations would not have been appropriate.
I just wanted to point out a type of situation from my experience that goes against your generalization, to give a more complete picture.
- m_mueller 10y agoThanks, I'll consider clojure in the future. How would you say its standard lib and third party libraries hold up against python? With my point about the need to plan ahead I specifically meant python by the way, and I think all runtimes that can't make use of concurrent multithreading have this in common. As soon as you can do multithreading with shared memory you become a lot more flexible in parallelising later on, and you can adjust your code style accordingly to accommodate this without much overhead. That's why the GIL IMO is so unfortunate, it's a real shame that Python 3 wasn't able to get rid of this when it started with a clean slate.
- bmh100 10y agoI see now what you meant. I apologize for my misunderstanding in that to be a general statement. Knowing that, I agree with your statements much more now. From what I can tell, the multi-process architecture in Python would require everything you mentioned. Not removing the GIL is indeed unfortunate. Clojure is niche in a way. It is a Lisp dialect. However, it is based on the JVM and has built-in conveniences for using Java functions, classes, and libraries. Therefore, comparing libraries for enterprise applications is more accurately Clojure+Java vs. Python. My expertise is not in either language ecosystem, but my impression is that Python has better libraries for data analysis and as a system scripting language, and Clojure+Java is better for general applications and everything else. One benefit of Clojure that makes parallelization easy is the immutable data structures. New data structures are cheaply made by seamlessly referencing originating data structures. Example: making a copy of array A and appending an item as array B, means that array B is mostly a reference to array A. Immutable data structures also offer expectations of behavior that mutable state naturally offends, such as avoiding race conditions or locks. In the case of my earlier example, I simply added some "future" wrapper functions and "dereference" annotations [1]. The entire program would execute to reach the final output, then work in reverse order to trigger and plan the future functions. Order of execution was based on dependencies, not strictly on the order of the code. It was fascinating was to watch the logs for when functions would start or complete. If you are familiar with operations planning and critical path analysis, the overall run-time reflected the critical path. [1]: https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/clojure.core/future https://www.conj.io/store/v1/org.clojure/clojure/1.8.0/clj/c...