4 ms·

Ok, I should have said that I am only interested in the "multiprocessing" aspect; That sounds pretty promising what you write about MP in Clojure, although. Con

by fnl 16y ago

Ok, I should have said that I am only interested in the "multiprocessing" aspect; That sounds pretty promising what you write about MP in Clojure, although. Concerning Hadoop, I have my doubts about it - I never switched and now I even read people are going back to SQL & OpenMP (or MPI) anyways. Python's "old wound" GIL story is "solved" (let's say, "patched" ;) ) since 2.6 (if you can accept the huge overhead from multiprocessing's calls for coarse-grained threads), so I am fine with that so far. So, has anybody got insight into how Clojure (or even pvmap/pvreduce specifically) performs against either OpenMP/MPI or Python? EDIT: I maybe better should say, how much easier it is to use. After, all it is a JVM, so it will be slower than C, obviously, but you also get rid of the million of possible bugs you can introduce in it (which makes using Python so attractive to me). So, is there a good reason for somebody who as used either Python or OpenMP for multiprocessing to switch to Clojure?

leif 16y ago
Python is almost always memory-heavy. C is not, if you do it right. Of course, C, if you do it right, will be the fastest, it's the "do it right" that's hard. Clojure, on the JVM, will be slower than C "done right", but faster than most C, because the JVM's JIT is surprisingly smart. Any performance problems you get with Clojure can usually be mitigated by adding type hints to force your repetitive calculations to use primitive type operations, which will get compiled by the JVM to direct machine instructions, as fast as C. You should switch to Clojure from Python because it's faster. You should switch to Clojure from C because it's easier, and more specifically, easier to get it to a certain level of speed (which is high). Don't let the JVM fool you into thinking it's slow. It isn't.
- fnl 16y ago
  Well, I do not quite agree with your argument that Python is "always memory-hungry". As a matter of fact, writing "good" Python code is a very seldom observed art and most performance issues can be gotten rid of if you know enough about the language and follow the style and design advices of true "Pythonistas", as the "gurus" there call themselves (and many more who do not deserve that title ;)). However, I have been programming several years in Java on an earlier assignment, where stuff got so clumsy I wrote a 6 page manual for my team outlining "don't do's" in Java because it would either cost tons of performance or memory, mostly related to very basic stuff such as instantiation strategies and everything that might be "good" for OOD, but bad for performance. We ended up writing most of the core of the system in Lisp and then generated the "light-weight" Java stuff from a DSL one of our Lisp "masters" had conjured, to avoid all the bloat you have to go through to write simple things such as walkers, etc. in Java. If I ever had real memory problems ever, I must say it was with JVM, and it took me months to, for example, come up with a solution to index a few TBs with Lucene and the JVM "crashing" (well, hanging) because of gc (although that was back more than five years now, so things will have gotten better, I admit). Nay, I must say the Python/C combo is much to my liking. I want something that fits into that picture, not so much an alternative. And there I currently see Erlang as quite promising combo to my two favorite languages, after going through all the advice and pointers everybody here gave me.
  - leif 16y ago
    > As a matter of fact, writing "good" Python code is a very seldom observed art and most performance issues can be gotten rid of if you know enough about the language and follow the style and design advices of true "Pythonistas", as the "gurus" there call themselves (and many more who do not deserve that title ;)). First of all, should performant code in your language be "seldom observed" and require the knowledge of the "gurus"? I should hope not. I have written plenty of optimized python in my day, and it's always been a decent constant behind c or java, no matter how many of the "best practices" I follow. Pypy may be a different story, but their toolchain was too complex and slow last I looked. Most of python's best practices are language bugs, anyway. Generators are great, but most of the rest is crap. I should cache my namespace lookups outside the loop? Why can't you do that for me, Python? That's one of the simplest compiler optimizations we have. All the fastest Python is written in C extensions. Why do you think this is?
    - fnl 16y ago
      As an update: Python 3k solves the namespace issue you mention. (And I hope more and more dev's will finally move on, I love it and only can say it is 100% worth the switch! I'll never again code in pre-3k if not forced to.) And no, I am not saying that Python outperforms Java, obviously. But it can provide very decent performance at very low LOCs. And then you can identify the bottlenecks and implement that part in C. Overall, a much faster approach than writing in Java (and, obviously, than writing in pure C) with final performance rates beating a JVM easily. EDIT: No, that is actually even too complicated than what I mostly need to do: For the real bottlenecks I usually just need to find the C library that solves that hard part (e.g., some number-crunching stuff I am doing) and already has a Python wrapper, or you generate one (hello SWIG), or, worst case, you really have to write one (yet have to find that case, although).

by fnl 16y ago

leif 16y ago

Python is almost always memory-heavy. C is not, if you do it right. Of course, C, if you do it right, will be the fastest, it's the "do it right" that's hard. Clojure, on the JVM, will be slower than C "done right", but faster than most C, because the JVM's JIT is surprisingly smart. Any performance problems you get with Clojure can usually be mitigated by adding type hints to force your repetitive calculations to use primitive type operations, which will get compiled by the JVM to direct machine instructions, as fast as C. You should switch to Clojure from Python because it's faster. You should switch to Clojure from C because it's easier, and more specifically, easier to get it to a certain level of speed (which is high). Don't let the JVM fool you into thinking it's slow. It isn't.

fnl 16y ago
Well, I do not quite agree with your argument that Python is "always memory-hungry". As a matter of fact, writing "good" Python code is a very seldom observed art and most performance issues can be gotten rid of if you know enough about the language and follow the style and design advices of true "Pythonistas", as the "gurus" there call themselves (and many more who do not deserve that title ;)). However, I have been programming several years in Java on an earlier assignment, where stuff got so clumsy I wrote a 6 page manual for my team outlining "don't do's" in Java because it would either cost tons of performance or memory, mostly related to very basic stuff such as instantiation strategies and everything that might be "good" for OOD, but bad for performance. We ended up writing most of the core of the system in Lisp and then generated the "light-weight" Java stuff from a DSL one of our Lisp "masters" had conjured, to avoid all the bloat you have to go through to write simple things such as walkers, etc. in Java. If I ever had real memory problems ever, I must say it was with JVM, and it took me months to, for example, come up with a solution to index a few TBs with Lucene and the JVM "crashing" (well, hanging) because of gc (although that was back more than five years now, so things will have gotten better, I admit). Nay, I must say the Python/C combo is much to my liking. I want something that fits into that picture, not so much an alternative. And there I currently see Erlang as quite promising combo to my two favorite languages, after going through all the advice and pointers everybody here gave me.
- leif 16y ago
  > As a matter of fact, writing "good" Python code is a very seldom observed art and most performance issues can be gotten rid of if you know enough about the language and follow the style and design advices of true "Pythonistas", as the "gurus" there call themselves (and many more who do not deserve that title ;)). First of all, should performant code in your language be "seldom observed" and require the knowledge of the "gurus"? I should hope not. I have written plenty of optimized python in my day, and it's always been a decent constant behind c or java, no matter how many of the "best practices" I follow. Pypy may be a different story, but their toolchain was too complex and slow last I looked. Most of python's best practices are language bugs, anyway. Generators are great, but most of the rest is crap. I should cache my namespace lookups outside the loop? Why can't you do that for me, Python? That's one of the simplest compiler optimizations we have. All the fastest Python is written in C extensions. Why do you think this is?
  - fnl 16y ago
    As an update: Python 3k solves the namespace issue you mention. (And I hope more and more dev's will finally move on, I love it and only can say it is 100% worth the switch! I'll never again code in pre-3k if not forced to.) And no, I am not saying that Python outperforms Java, obviously. But it can provide very decent performance at very low LOCs. And then you can identify the bottlenecks and implement that part in C. Overall, a much faster approach than writing in Java (and, obviously, than writing in pure C) with final performance rates beating a JVM easily. EDIT: No, that is actually even too complicated than what I mostly need to do: For the real bottlenecks I usually just need to find the C library that solves that hard part (e.g., some number-crunching stuff I am doing) and already has a Python wrapper, or you generate one (hello SWIG), or, worst case, you really have to write one (yet have to find that case, although).