4 ms·

Tuning Concurrent Clojure

43 points by mreid 17y ago

jimbokun 17y ago
I'm really enjoying this series. Definitely getting down and gritty with what Clojure is really capable of (and all the other languages he's tried). Of all the articles I've read on concurrent programming and the languages that promise new ways to handle it, I find the Wide Finder series the most pragmatic and concrete.
- swannodette 17y ago
  I wouldn't say that the article gets down to the nitty gritty of what Clojure is capable at all. It's a great series about a high profile hacker having a go at what I think is a great new programming language. For a down and gritty analysis of Clojure optimization you're better served reading Christophe Grand's posts. Or even my own fast but butt ugly optimization of Uncle Bob's Jarvis March- http://github.com/swannodette/convex-hull/blob/master/convex_hull.clj http://github.com/swannodette/convex-hull/blob/master/convex... where I use macros to inline everything. His version ran in 25s, mine in about 0.5.
icey 17y ago
I'd really like to see a Common Lisp solution to the wide finder sometime (in case any lisper is really bored and feels like cranking one out :D )
anonjon 17y ago
I am shaking my head in dismay. Reduce is a loop. Rewriting all of your loop code as reduce won't get you any performance gains. If anything you will lose performance over looping because you are now passing stuff to a function and it is getting boxed. Why is he splitting all of the lines with a #"\n" regex? there are perfectly good methods for reading something line-by-line in a number of places (in both java and clojure). I like Tim Bray and anyone who writes about clojure or lisp, but I feel like I'm tripping on acid here.
- runevault 17y ago
  Reduce is actually faster than using loop/recur normally IME. Probably at least in part due to chuncked seqs which map and reduce among other functions can take advantage of. Most of the built in clojure functions are better optimized than doing things by hand yourself.
  - anonjon 17y ago
    Reduce /IS/ a loop: (from clojure.core) ([f val coll] (let [s (seq coll)] (if s (if (chunked-seq? s) (recur f (.reduce (chunk-first s) f val) (chunk-next s)) (recur f (f val (first s)) (next s))) val))) It follows that I can use chunks in a loop... chunking and looping are not mutually exclusive. It also happens to be the case that reduce is a function call, and (AFAIK) that you can't use transients (like conj!) or make use of unboxed values. This is an issue, as such, one should still loop for best performance.
    - swannodette 17y ago
      If I'm not mistaken chunking is only supported on some of Clojure's data structures. Reduce is certainly the better option with vectors (and maps?) if you write your own loop/recur you'll simply be duplicating the existing implementation. Also I really can't imagine using loop/recur on anything except for vectors and maps most of the time anyway, so why not just use reduce?
mcav 17y ago
As a Python programmer who has heretofore found Clojure very favorably designed, this is a little disheartening. I haven't programmed in a functional language yet; Python has served me really well for most everything, and C/C++ has always covered gaps where I need a little speed boost. Yet I've always seen functional languages as a real fundamental approach that seems really elegant in theory. I have "Programming Clojure" on my Christmas list. Something about functional languages' elegance makes me really want to have a functional language succeed long-term. But every single functional language I've seen so far seems to miss the mark in seemingly important ways: Erlang with its syntax and purported string issues; Common Lisp (et. al) with its fragmented variants (so I hear); etc. I would really like to see a great functional language executed well. You know, the whole deal: speed, interoperability, diverse libraries, concurrency... Maybe Clojure is it, and it just has warts to iron out. Maybe the obtuseness that Tim Bray described here is just not "the way to do it". I hope that functional languages aren't inherently more difficult to work with than, say, Python. I haven't worked with any of these functional languages yet, so granted I probably shouldn't be complaining. I know that with more experience in the languages I'd probably get a better understanding of how they excel -- and, unfortunately, where they don't. Just... nothing seems to hit the sweet spot as well as Python and Javascript has, for me. I guess that's an unrealistic expectation for not having actually worked with functional languages. I want a functional language to feel as smooth as Python. (If that just requires more experience and familiarity, fine. But I'd be hard-pressed to believe that the code demonstrated here could seem as clear to a Lisp expert as Python equivalents would feel to a Python programmer.) Are the issues Tim found in this post actually just caused by lack of experience, or is this really how the language gets down to feel? I'd love to hear what lisp (clojure or not) folks can share.
- swannodette 17y ago
  I would say Tim Bray is making an amazing effort to tackle the hard problem of optimizing a language that he has even at this point only passing familiarity. That said, it's great to have such a well known hacker try to put Clojure through it's paces. Having optimized my fair share of Clojure programs it's certainly possible to get very close to Java speeds. Also in my experience there's often a point of diminishing return. It's often simpler to just write that tight loop in a tiny Java class and be done with it. Just as Python is extensible in C when you need performance, Clojure is extensible in Java. With the work in the Clojure new branch, the possibility of writing wicked fast Clojure in Clojure is just around the corner. I work in Python and JavaScript day in and day out and my comparisons follow: I can say without a doubt, idiomatic Clojure (not heavily optimized) is generally faster than Python, and more elegant IMHO. Once you get used to pretty much everything returning a value, Python can drive you insane. The only thing I generally miss about Python is that's more usefulf/friendly as a command line scripting language. But for building an app, Clojure all the way. I use JavaScript even more, and Clojure has all the things I like about JavaScript except even more flexible and useful in contexts other than web programming (OpenGL programming for example). I picked up Clojure precisely because I wanted a fast elegant dynamic and concurrency ready programming language. As the JDK evolves and our laptops start having 4 cores, I think the case for Clojure will only grow.

43 points by mreid 17y ago

I'm really enjoying this series. Definitely getting down and gritty with what Clojure is really capable of (and all the other languages he's tried). Of all the articles I've read on concurrent programming and the languages that promise new ways to handle it, I find the Wide Finder series the most pragmatic and concrete.

swannodette 17y ago
I wouldn't say that the article gets down to the nitty gritty of what Clojure is capable at all. It's a great series about a high profile hacker having a go at what I think is a great new programming language. For a down and gritty analysis of Clojure optimization you're better served reading Christophe Grand's posts. Or even my own fast but butt ugly optimization of Uncle Bob's Jarvis March- http://github.com/swannodette/convex-hull/blob/master/convex_hull.clj http://github.com/swannodette/convex-hull/blob/master/convex... where I use macros to inline everything. His version ran in 25s, mine in about 0.5.

icey 17y ago

I'd really like to see a Common Lisp solution to the wide finder sometime (in case any lisper is really bored and feels like cranking one out :D )

anonjon 17y ago

I am shaking my head in dismay. Reduce is a loop. Rewriting all of your loop code as reduce won't get you any performance gains. If anything you will lose performance over looping because you are now passing stuff to a function and it is getting boxed. Why is he splitting all of the lines with a #"\n" regex? there are perfectly good methods for reading something line-by-line in a number of places (in both java and clojure). I like Tim Bray and anyone who writes about clojure or lisp, but I feel like I'm tripping on acid here.

runevault 17y ago
Reduce is actually faster than using loop/recur normally IME. Probably at least in part due to chuncked seqs which map and reduce among other functions can take advantage of. Most of the built in clojure functions are better optimized than doing things by hand yourself.
- anonjon 17y ago
  Reduce /IS/ a loop: (from clojure.core) ([f val coll] (let [s (seq coll)] (if s (if (chunked-seq? s) (recur f (.reduce (chunk-first s) f val) (chunk-next s)) (recur f (f val (first s)) (next s))) val))) It follows that I can use chunks in a loop... chunking and looping are not mutually exclusive. It also happens to be the case that reduce is a function call, and (AFAIK) that you can't use transients (like conj!) or make use of unboxed values. This is an issue, as such, one should still loop for best performance.
  - swannodette 17y ago
    If I'm not mistaken chunking is only supported on some of Clojure's data structures. Reduce is certainly the better option with vectors (and maps?) if you write your own loop/recur you'll simply be duplicating the existing implementation. Also I really can't imagine using loop/recur on anything except for vectors and maps most of the time anyway, so why not just use reduce?

mcav 17y ago

As a Python programmer who has heretofore found Clojure very favorably designed, this is a little disheartening. I haven't programmed in a functional language yet; Python has served me really well for most everything, and C/C++ has always covered gaps where I need a little speed boost. Yet I've always seen functional languages as a real fundamental approach that seems really elegant in theory. I have "Programming Clojure" on my Christmas list. Something about functional languages' elegance makes me really want to have a functional language succeed long-term. But every single functional language I've seen so far seems to miss the mark in seemingly important ways: Erlang with its syntax and purported string issues; Common Lisp (et. al) with its fragmented variants (so I hear); etc. I would really like to see a great functional language executed well. You know, the whole deal: speed, interoperability, diverse libraries, concurrency... Maybe Clojure is it, and it just has warts to iron out. Maybe the obtuseness that Tim Bray described here is just not "the way to do it". I hope that functional languages aren't inherently more difficult to work with than, say, Python. I haven't worked with any of these functional languages yet, so granted I probably shouldn't be complaining. I know that with more experience in the languages I'd probably get a better understanding of how they excel -- and, unfortunately, where they don't. Just... nothing seems to hit the sweet spot as well as Python and Javascript has, for me. I guess that's an unrealistic expectation for not having actually worked with functional languages. I want a functional language to feel as smooth as Python. (If that just requires more experience and familiarity, fine. But I'd be hard-pressed to believe that the code demonstrated here could seem as clear to a Lisp expert as Python equivalents would feel to a Python programmer.) Are the issues Tim found in this post actually just caused by lack of experience, or is this really how the language gets down to feel? I'd love to hear what lisp (clojure or not) folks can share.

swannodette 17y ago
I would say Tim Bray is making an amazing effort to tackle the hard problem of optimizing a language that he has even at this point only passing familiarity. That said, it's great to have such a well known hacker try to put Clojure through it's paces. Having optimized my fair share of Clojure programs it's certainly possible to get very close to Java speeds. Also in my experience there's often a point of diminishing return. It's often simpler to just write that tight loop in a tiny Java class and be done with it. Just as Python is extensible in C when you need performance, Clojure is extensible in Java. With the work in the Clojure new branch, the possibility of writing wicked fast Clojure in Clojure is just around the corner. I work in Python and JavaScript day in and day out and my comparisons follow: I can say without a doubt, idiomatic Clojure (not heavily optimized) is generally faster than Python, and more elegant IMHO. Once you get used to pretty much everything returning a value, Python can drive you insane. The only thing I generally miss about Python is that's more usefulf/friendly as a command line scripting language. But for building an app, Clojure all the way. I use JavaScript even more, and Clojure has all the things I like about JavaScript except even more flexible and useful in contexts other than web programming (OpenGL programming for example). I picked up Clojure precisely because I wanted a fast elegant dynamic and concurrency ready programming language. As the JDK evolves and our laptops start having 4 cores, I think the case for Clojure will only grow.