4 ms·

The other way you have to leave the Clojure ecosystem is when you have to communicate across the network or persist to disk. Writing everything fits in memory d

The other way you have to leave the Clojure ecosystem is when you have to communicate across the network or persist to disk. Writing everything fits in memory data analysis in Clojure is some of the cleanest and most fun programming I've ever done. But once you want to spread between multiple computers or operate on data larger than what comfortably fits in memory, you end up coding just like you would in C#. I started working on some versions of persistent data structures that serialize to a KV store, which lets you send a hash map to another machine, have save that machine add a key, then send it back. It seemed to work pretty well, but there's a ~10x performance penalty over Clojure's inbuilt data types (which are themselves not lighting the world on fire speed wise). That's a pretty harsh price to pay - you can use 100 machines to simulate single address space Clojure on a 10x machine. Awesome, but likely impractical. If you need the persistence between program runs or would like to run incremental computation on large data I could see it working well, though.

reitzensteinm 7y ago
So this got 3 downvotes, and I don't care about the karma, but I'd be quite curious to hear about opposing experiences scaling up Clojure programs since there seems to be some disagreement. If you started to struggle with fitting everything in one node due to CPU or memory constraints, but still felt like you were writing Clojure in a half way idiomatic style when moving to a cluster, I'd love to hear from you.
- SaxonRobber 7y ago
  That’s just what happens around here when you imply that a Lisp might not be perfect. Don’t think to hard about it, they don’t either.
- stonemetal12 7y ago
  ~10X a performance penalty doesn't seem that bad. In C a pointer derefference will beat a web service call by more than 10X. Also why a handjammed KV store? EDN and other(faster) serialization libraries have been around for quite a while.
  - reitzensteinm 7y ago
    I used nippy originally, then later Java serialization libraries once it became clear optimization was required. I tried using both Redis and RocksDB. The happy case had all but the last layer of nodes sitting in cache in memory (as long as your memory is more than 1/32 of the size of your disk). I think I could do better starting from scratch given what I've learned.
emidln 7y ago
When I was using HoneySQL[0] speaking to PostgreSQL over JDBC to deal with persisted data, I did not feel non-idiomatic. When I used Kafka and Onyx[1] to process large dataflows, I did not feel non-idiomatic. Granted my working set was only a few 10s of TB, but there wasn't much pain involved. I tended to use Nippy[1] for serialization. I didn't use special data structures (except when the problem called for it), with a clear separation using nippy serialization (in place of pr-str/read-string) at the edges of my app. [0] https://github.com/jkk/honeysql https://github.com/jkk/honeysql [1] http://www.onyxplatform.org/ http://www.onyxplatform.org/ [2] https://github.com/ptaoussanis/nippy https://github.com/ptaoussanis/nippy
- reitzensteinm 7y ago
  But did you really get much of the benefit of using Clojure? If you store your data in an SQL database for instance, that layer provides the equivalent of what Clojure's STM can do within the process. You open a transaction, run some queries, perform some logic, then update the database. Clojure's persistent data structures don't give you extra leverage here - you could just as well write the logic with mutable datastructures in C# (that you throw away afterwards). This is no accident, as Clojure's STM was built around the premise of emulating the MVCC provided by databases to provide ACI (minus the D) guarantees to program against. If you're already getting those guarantees elsewhere, doubling up does no good. Similarly, code that operates on streams is generally not where you get bogged down by state and multithreading in something like C#. I say this as somebody that believes Clojure is a powerful addition to the toolbox. I even use refs in production code...
  - robto 7y ago
    I say you still do get a benefit from using because of the repl-driven development process. I miss that instantaneous feedback and the feeling that you are in dialogue with your program whenever I develop in other languages. Add to that the fact that the language shepards you into using immutable datastructures and writing pure code, I still think Clojure is a significant advantage when writing distributed applications.
  - emidln 7y ago

reitzensteinm 7y ago

So this got 3 downvotes, and I don't care about the karma, but I'd be quite curious to hear about opposing experiences scaling up Clojure programs since there seems to be some disagreement. If you started to struggle with fitting everything in one node due to CPU or memory constraints, but still felt like you were writing Clojure in a half way idiomatic style when moving to a cluster, I'd love to hear from you.

SaxonRobber 7y ago
That’s just what happens around here when you imply that a Lisp might not be perfect. Don’t think to hard about it, they don’t either.
stonemetal12 7y ago
~10X a performance penalty doesn't seem that bad. In C a pointer derefference will beat a web service call by more than 10X. Also why a handjammed KV store? EDN and other(faster) serialization libraries have been around for quite a while.
- reitzensteinm 7y ago
  I used nippy originally, then later Java serialization libraries once it became clear optimization was required. I tried using both Redis and RocksDB. The happy case had all but the last layer of nodes sitting in cache in memory (as long as your memory is more than 1/32 of the size of your disk). I think I could do better starting from scratch given what I've learned.

emidln 7y ago

When I was using HoneySQL[0] speaking to PostgreSQL over JDBC to deal with persisted data, I did not feel non-idiomatic. When I used Kafka and Onyx[1] to process large dataflows, I did not feel non-idiomatic. Granted my working set was only a few 10s of TB, but there wasn't much pain involved. I tended to use Nippy[1] for serialization. I didn't use special data structures (except when the problem called for it), with a clear separation using nippy serialization (in place of pr-str/read-string) at the edges of my app. [0] https://github.com/jkk/honeysql https://github.com/jkk/honeysql [1] http://www.onyxplatform.org/ http://www.onyxplatform.org/ [2] https://github.com/ptaoussanis/nippy https://github.com/ptaoussanis/nippy

reitzensteinm 7y ago
But did you really get much of the benefit of using Clojure? If you store your data in an SQL database for instance, that layer provides the equivalent of what Clojure's STM can do within the process. You open a transaction, run some queries, perform some logic, then update the database. Clojure's persistent data structures don't give you extra leverage here - you could just as well write the logic with mutable datastructures in C# (that you throw away afterwards). This is no accident, as Clojure's STM was built around the premise of emulating the MVCC provided by databases to provide ACI (minus the D) guarantees to program against. If you're already getting those guarantees elsewhere, doubling up does no good. Similarly, code that operates on streams is generally not where you get bogged down by state and multithreading in something like C#. I say this as somebody that believes Clojure is a powerful addition to the toolbox. I even use refs in production code...
- robto 7y ago
  I say you still do get a benefit from using because of the repl-driven development process. I miss that instantaneous feedback and the feeling that you are in dialogue with your program whenever I develop in other languages. Add to that the fact that the language shepards you into using immutable datastructures and writing pure code, I still think Clojure is a significant advantage when writing distributed applications.
- emidln 7y ago