4 ms·

These are two questions (see the recently posted parallel clojure talk). For concurrency (short version: this means multiple processes doing different things s

• fnl 16y ago
  Ok, I should have said that I am only interested in the "multiprocessing" aspect; That sounds pretty promising what you write about MP in Clojure, although. Concerning Hadoop, I have my doubts about it - I never switched and now I even read people are going back to SQL & OpenMP (or MPI) anyways. Python's "old wound" GIL story is "solved" (let's say, "patched" ;) ) since 2.6 (if you can accept the huge overhead from multiprocessing's calls for coarse-grained threads), so I am fine with that so far. So, has anybody got insight into how Clojure (or even pvmap/pvreduce specifically) performs against either OpenMP/MPI or Python? EDIT: I maybe better should say, how much easier it is to use. After, all it is a JVM, so it will be slower than C, obviously, but you also get rid of the million of possible bugs you can introduce in it (which makes using Python so attractive to me). So, is there a good reason for somebody who as used either Python or OpenMP for multiprocessing to switch to Clojure?
  • leif 16y ago
    Python is almost always memory-heavy. C is not, if you do it right. Of course, C, if you do it right, will be the fastest, it's the "do it right" that's hard. Clojure, on the JVM, will be slower than C "done right", but faster than most C, because the JVM's JIT is surprisingly smart. Any performance problems you get with Clojure can usually be mitigated by adding type hints to force your repetitive calculations to use primitive type operations, which will get compiled by the JVM to direct machine instructions, as fast as C. You should switch to Clojure from Python because it's faster. You should switch to Clojure from C because it's easier, and more specifically, easier to get it to a certain level of speed (which is high). Don't let the JVM fool you into thinking it's slow. It isn't.
    • fnl 16y ago
      Well, I do not quite agree with your argument that Python is "always memory-hungry". As a matter of fact, writing "good" Python code is a very seldom observed art and most performance issues can be gotten rid of if you know enough about the language and follow the style and design advices of true "Pythonistas", as the "gurus" there call themselves (and many more who do not deserve that title ;)). However, I have been programming several years in Java on an earlier assignment, where stuff got so clumsy I wrote a 6 page manual for my team outlining "don't do's" in Java because it would either cost tons of performance or memory, mostly related to very basic stuff such as instantiation strategies and everything that might be "good" for OOD, but bad for performance. We ended up writing most of the core of the system in Lisp and then generated the "light-weight" Java stuff from a DSL one of our Lisp "masters" had conjured, to avoid all the bloat you have to go through to write simple things such as walkers, etc. in Java. If I ever had real memory problems ever, I must say it was with JVM, and it took me months to, for example, come up with a solution to index a few TBs with Lucene and the JVM "crashing" (well, hanging) because of gc (although that was back more than five years now, so things will have gotten better, I admit). Nay, I must say the Python/C combo is much to my liking. I want something that fits into that picture, not so much an alternative. And there I currently see Erlang as quite promising combo to my two favorite languages, after going through all the advice and pointers everybody here gave me.
      • leif 16y ago
        > As a matter of fact, writing "good" Python code is a very seldom observed art and most performance issues can be gotten rid of if you know enough about the language and follow the style and design advices of true "Pythonistas", as the "gurus" there call themselves (and many more who do not deserve that title ;)). First of all, should performant code in your language be "seldom observed" and require the knowledge of the "gurus"? I should hope not. I have written plenty of optimized python in my day, and it's always been a decent constant behind c or java, no matter how many of the "best practices" I follow. Pypy may be a different story, but their toolchain was too complex and slow last I looked. Most of python's best practices are language bugs, anyway. Generators are great, but most of the rest is crap. I should cache my namespace lookups outside the loop? Why can't you do that for me, Python? That's one of the simplest compiler optimizations we have. All the fastest Python is written in C extensions. Why do you think this is?