4 ms·

This is fascinating, but for a lot of us building web apps we have to scale beyond a single box, which means shared state needs to be in something like memcache

This is fascinating, but for a lot of us building web apps we have to scale beyond a single box, which means shared state needs to be in something like memcached. I love Clojure, but it's concurrency constructs aren't that relevant to most web programming. Which is fine, but let's keep in mind that developing a web app with Clojure is painful now due to a lack of libraries / a large enough community. I'm eagerly awaiting the day that changes.

frognibble 16y ago
There are a few usable libraries for building web apps in Clojure (Compojure is an example). These libraries are not as comprehensive as libraries in other languages and the communities are small as you noted. That said, I wouldn't describe the experience as painful. Some of the libraries are pleasant to use. I agree with your comment regarding Clojure's concurrency constructs. I am building my web app to run on more than one machine. I have not yet had a need to use any of the concurrency constructs.
cageface 16y ago
I'm still trying to figure out exactly where the Clojure concurrency model is useful. How many problem domains require extremely high throughput in a single shared memory space? It seems to me that you're usually either well within the bounds of what a single cpu can handle or you're going to need multiple machines and can just keep throwing procs at the problem.
- troutwine 16y ago
  This echoes my sentiments rather precisely. On the not infrequent occasion that I am building something which requires massive concurrency I _also_ need it to be distributed, if only for service reliability. Multi-core might be the (very near) future, but distributed processing is now. Languages that make distributed computing easy (well, easier) are more valuable, to my mind, than those that make multi-core computing more simple. That is why, for me, Clojure is a lovely little toy, but Erlang's a workhorse.
  - dedward 16y ago
    Those are still different issues - threading & concurrency in one memory area -vs- distributed computing. Clojure is attempting to optimize for the former... it's easy to write thread-safe applications.
    - troutwine 16y ago
      They are distinct issues, to be sure, but, then again, they are part and parcel. Properly implemented, a language with distributed computing primitives buys you single address space concurrency: it's the degenerate case of a cluster. No matter what, a single machine is going to be less in terms of total computational capacity and fail-over ability. Languages that optimize for single machines, however nice, are something of a tease.
- deleted 16y ago
  [deleted]
- jared314 16y ago
  In a previous post, he mentioned Swarmiji (http://github.com/amitrathore/swarmiji http://github.com/amitrathore/swarmiji) for spreading things across multiple machines.
- pjscott 16y ago
  It's great for doing things like k-means clustering or Delaunay triangulation. Those might not come up a lot for you, but some people need to do them, and would love to be able to speed that up by throwing several processor cores at the problem. Solving them on a cluster is a lot harder, and often unnecessary. (On k-means clustering, in particular, Clojure-style transactional memory gives an almost linear speedup with the number of processor cores, without significantly changing the code. That's worth something.)
  - kanak 16y ago
    Could you please provide a link to the implementation? I'm really interested in learning how this is done. Thanks.
    - pjscott 16y ago
      Sorry, I can't point to any code, but in both cases the approach is the same: you have some shared data structures, like an array of regions containing points, or a red-black tree, or a priority queue, and several threads all read and write these data structures. Since transactional memory lets you run concurrently unless there are actual memory conflicts, this often gives better concurrency than locks. Of course, for examples like these, the tricky part is designing data structures that don't have many inherent memory conflicts. Red-black trees, for example, are tricky because the rebalancing transformations tend to step on other threads' toes.
- jerf 16y ago
  Counter-question: How many people have been so trained by decades of terror with multithreading horrors that they can no longer see when it is the right choice when you have a non-nightmare multithreading capability? When you encounter a new paradigm like this, its very important to work in it for at least a little and give yourself some time to untrain before writing off the usefulness of the paradigm. I work mostly in servers and the answer to your question is really quite a lot. (Unfortunately, I'm in the worst place, I untrained myself and am now able to see where this is useful, but not really able to use it at work. Sad.) I am reminded of this I saw yesterday: http://www.reddit.com/r/programming/comments/cq6wo/why_dvcs_provides_better_central_control/c0uen0a http://www.reddit.com/r/programming/comments/cq6wo/why_dvcs_...
JulianMorrison 16y ago
Clojure + terracotta should work well for small to medium scale-out of shared data.
- kanak 16y ago
  Has this been done by someone? I remember the same answer (use Clojure + Terracotta) being given months ago when someone in the clojure google group asked for erlang type distribution support. I was wondering if someone had actually done it to see how well it works.
  - bnoordhuis 16y ago
    I can answer this question for Java + Terracotta. Terracotta works much of its magic through bytecode instrumentation so I expect that the differences between Clojure and Java are minimal. Pros: 1. Makes it very easy to turn an existing threaded application into a multi-node application, often no code changes required. Killer feature. 2. Integrates seamlessly with Spring and Ehcache. Perhaps less relevant for Clojure developers but a big boon for JEE people. Cons: 1. It uses multicast for replication. This is an implementation detail, not a conceptual drawback, but it's a pain to set up when your machines live in different network segments. 2. Can potentially transmit stupid amounts of data over the wire. I suppose this is part design, part configuration.
    - JulianMorrison 16y ago
      If you are going to use Terracotta you really do need to design for it - even if it could in theory wrap a naive design, you'll probably thrash the network. Some ideas for Terracotta friendly design I can think of, right off the bat: 1. If this node is the only one using this particular shared data structure right now, it gets to operate locally without network transactions until it's done. 2. If you consistently use a small part of a large data structure (eg: a hash), you'll only have that small part "paged in" on that one node. The whole data structure can be larger than RAM. Thus, node affinity can be important. 3. Be careful how much work is done inside a synchronized method of a shared data structure. It should be a medium amount. Too much, and you'll make other users wait, too little and you'll thrash the network.

frognibble 16y ago

There are a few usable libraries for building web apps in Clojure (Compojure is an example). These libraries are not as comprehensive as libraries in other languages and the communities are small as you noted. That said, I wouldn't describe the experience as painful. Some of the libraries are pleasant to use. I agree with your comment regarding Clojure's concurrency constructs. I am building my web app to run on more than one machine. I have not yet had a need to use any of the concurrency constructs.

cageface 16y ago

I'm still trying to figure out exactly where the Clojure concurrency model is useful. How many problem domains require extremely high throughput in a single shared memory space? It seems to me that you're usually either well within the bounds of what a single cpu can handle or you're going to need multiple machines and can just keep throwing procs at the problem.

troutwine 16y ago
This echoes my sentiments rather precisely. On the not infrequent occasion that I am building something which requires massive concurrency I _also_ need it to be distributed, if only for service reliability. Multi-core might be the (very near) future, but distributed processing is now. Languages that make distributed computing easy (well, easier) are more valuable, to my mind, than those that make multi-core computing more simple. That is why, for me, Clojure is a lovely little toy, but Erlang's a workhorse.
- dedward 16y ago
  Those are still different issues - threading & concurrency in one memory area -vs- distributed computing. Clojure is attempting to optimize for the former... it's easy to write thread-safe applications.
  - troutwine 16y ago
    They are distinct issues, to be sure, but, then again, they are part and parcel. Properly implemented, a language with distributed computing primitives buys you single address space concurrency: it's the degenerate case of a cluster. No matter what, a single machine is going to be less in terms of total computational capacity and fail-over ability. Languages that optimize for single machines, however nice, are something of a tease.
deleted 16y ago
[deleted]
jared314 16y ago
In a previous post, he mentioned Swarmiji (http://github.com/amitrathore/swarmiji http://github.com/amitrathore/swarmiji) for spreading things across multiple machines.
pjscott 16y ago
It's great for doing things like k-means clustering or Delaunay triangulation. Those might not come up a lot for you, but some people need to do them, and would love to be able to speed that up by throwing several processor cores at the problem. Solving them on a cluster is a lot harder, and often unnecessary. (On k-means clustering, in particular, Clojure-style transactional memory gives an almost linear speedup with the number of processor cores, without significantly changing the code. That's worth something.)
- kanak 16y ago
  Could you please provide a link to the implementation? I'm really interested in learning how this is done. Thanks.
  - pjscott 16y ago
    Sorry, I can't point to any code, but in both cases the approach is the same: you have some shared data structures, like an array of regions containing points, or a red-black tree, or a priority queue, and several threads all read and write these data structures. Since transactional memory lets you run concurrently unless there are actual memory conflicts, this often gives better concurrency than locks. Of course, for examples like these, the tricky part is designing data structures that don't have many inherent memory conflicts. Red-black trees, for example, are tricky because the rebalancing transformations tend to step on other threads' toes.
jerf 16y ago
Counter-question: How many people have been so trained by decades of terror with multithreading horrors that they can no longer see when it is the right choice when you have a non-nightmare multithreading capability? When you encounter a new paradigm like this, its very important to work in it for at least a little and give yourself some time to untrain before writing off the usefulness of the paradigm. I work mostly in servers and the answer to your question is really quite a lot. (Unfortunately, I'm in the worst place, I untrained myself and am now able to see where this is useful, but not really able to use it at work. Sad.) I am reminded of this I saw yesterday: http://www.reddit.com/r/programming/comments/cq6wo/why_dvcs_provides_better_central_control/c0uen0a http://www.reddit.com/r/programming/comments/cq6wo/why_dvcs_...

JulianMorrison 16y ago

Clojure + terracotta should work well for small to medium scale-out of shared data.

kanak 16y ago
Has this been done by someone? I remember the same answer (use Clojure + Terracotta) being given months ago when someone in the clojure google group asked for erlang type distribution support. I was wondering if someone had actually done it to see how well it works.
- bnoordhuis 16y ago
  I can answer this question for Java + Terracotta. Terracotta works much of its magic through bytecode instrumentation so I expect that the differences between Clojure and Java are minimal. Pros: 1. Makes it very easy to turn an existing threaded application into a multi-node application, often no code changes required. Killer feature. 2. Integrates seamlessly with Spring and Ehcache. Perhaps less relevant for Clojure developers but a big boon for JEE people. Cons: 1. It uses multicast for replication. This is an implementation detail, not a conceptual drawback, but it's a pain to set up when your machines live in different network segments. 2. Can potentially transmit stupid amounts of data over the wire. I suppose this is part design, part configuration.
  - JulianMorrison 16y ago
    If you are going to use Terracotta you really do need to design for it - even if it could in theory wrap a naive design, you'll probably thrash the network. Some ideas for Terracotta friendly design I can think of, right off the bat: 1. If this node is the only one using this particular shared data structure right now, it gets to operate locally without network transactions until it's done. 2. If you consistently use a small part of a large data structure (eg: a hash), you'll only have that small part "paged in" on that one node. The whole data structure can be larger than RAM. Thus, node affinity can be important. 3. Be careful how much work is done inside a synchronized method of a shared data structure. It should be a medium amount. Too much, and you'll make other users wait, too little and you'll thrash the network.