6 ms·

I got the Clojure bug at some point not long ago. Decided I’d write a crawler and some data munging stuff directly in Clojure since it’s all about data processi

I got the Clojure bug at some point not long ago. Decided I’d write a crawler and some data munging stuff directly in Clojure since it’s all about data processing. Crawlers naturally want to be stacks with pipelines and expressing them as tail recursions over URLs curried into transducers, etc was easy enough conceptually, by difficult in reality because you want crawlers to be stateful. I think in clojure you end up hacking state by just attaching stuff to the outputs of your functions and building of more complicated objects down the line. I discovered that stuff which is cake in SQL or a language with data frame support is often typically hard in Clojure. Eg eg joining data , aggregation, etc. The few clojure libraries for this sort of thing are complicated to use because they have a mechanical sympathy with how clojure executes. Eg https://github.com/nathanmarz/specter/ https://github.com/nathanmarz/specter/ So I wrote some of my own libraries to get over this hurdle. I then realised that there isn’t a single decent statistics library that still maintained, and that the java bindings were far from simple to us. At that point I gave up on clojure for my use case. That’s not to say that there isn’t something for which it’s awesone. But in my opinion it isn’t data processing.

ken 8y ago
I think the Clojure answer to stateful data storage is "use Datomic". There's a big impedance mismatch between the functional world and the stateful (RDBMS) world. Datomic extends the functional world all the way down to data storage. You can certainly use SQL, and it's not any worse than any other language (and in some ways a little better), but IME it's not Clojure-simple, either. It's still the most awkward part of all my Clojure programs. Clojure excels when you can avoid mutating state, and I think that's why people especially love it for web apps. It fits almost perfectly into that model.
- digitaLandscape 8y ago
  So clojure the language mainly exists as marketing for Datomic, the slow, expensive, proprietary, locked-in database solution? Eww.
weavejester 8y ago
Can you explain a little more why you think state is hard to manage in Clojure? I've done a fair amount of data processing in Clojure, and I'd consider it one of Clojure's strengths.
zmmmmm 8y ago
> I then realised that there isn’t a single decent statistics library that still maintained, and that the java bindings were far from simple to us I kind of found a similar problem with Scala. In theory yeah, you can use any Java library from Scala. But what was happening was I was continuously hitting the impedance mismatch between the scala collection and data types and the Java ones. Sometimes it was happening implicitly and doing automatic conversions of massive amounts of data without me knowing and I had to do all kinds of non-idiomatic stuff to get Scala to perform well. My Scala started off much prettier than my Groovy code usually is, but by the time I finished it was an abomination. At that point I gave up and went back to Groovy because while it lacks many of the functional features, idiomatic Groovy "just worked" and gave me most of what I was looking for in terms of a "better Java".
- vorg 8y ago
  > My Scala started off much prettier than my Groovy code usually is, but by the time I finished it was an abomination > Sometimes I think I should just use Kotlin, but then I would miss the ability to drop seamlessly into dynamic mode Have you considered Clojure for this use case? That might alleviate your problems with using Kotlin and Scala, without having to resort to a single-target language like Apache Groovy. After all, unlike what Clojure, Kotlin, and Scala all offer now, we're never going to see an edition of Groovy that targets Javascript, Android, or native CPU.
  - zmmmmm 8y ago
    I've always been intrigued by Clojure but it feels like a huge leap to dive into a fully Lisp-style, completely functional language. I guess single target isn't a particularly big problem for me; the JVM runs everywhere I need it to.
dm3 8y ago
For the data processing tasks my go-to tool is Manifold[1]. Depending on how the application is structured, you can keep state in atoms, the system map when using Component[2] or vars with Mount[3]. Manifold streams asynchronously join the state transformations together while allowing control over backpressure and threading. On the data analysis side things aren't rosy on the JVM. I'd delegate that part to something which actually has all the tools built-in, e.g. Julia/Python+Numpy/R. Apache Arrow[4] is a nice project to facilitate the dataframe interop. [1] https://github.com/ztellman/manifold https://github.com/ztellman/manifold [2] https://github.com/stuartsierra/component https://github.com/stuartsierra/component [3] https://github.com/tolitius/mount https://github.com/tolitius/mount [4] https://arrow.apache.org/ https://arrow.apache.org/

ken 8y ago

I think the Clojure answer to stateful data storage is "use Datomic". There's a big impedance mismatch between the functional world and the stateful (RDBMS) world. Datomic extends the functional world all the way down to data storage. You can certainly use SQL, and it's not any worse than any other language (and in some ways a little better), but IME it's not Clojure-simple, either. It's still the most awkward part of all my Clojure programs. Clojure excels when you can avoid mutating state, and I think that's why people especially love it for web apps. It fits almost perfectly into that model.

digitaLandscape 8y ago
So clojure the language mainly exists as marketing for Datomic, the slow, expensive, proprietary, locked-in database solution? Eww.

weavejester 8y ago

Can you explain a little more why you think state is hard to manage in Clojure? I've done a fair amount of data processing in Clojure, and I'd consider it one of Clojure's strengths.

zmmmmm 8y ago

> I then realised that there isn’t a single decent statistics library that still maintained, and that the java bindings were far from simple to us I kind of found a similar problem with Scala. In theory yeah, you can use any Java library from Scala. But what was happening was I was continuously hitting the impedance mismatch between the scala collection and data types and the Java ones. Sometimes it was happening implicitly and doing automatic conversions of massive amounts of data without me knowing and I had to do all kinds of non-idiomatic stuff to get Scala to perform well. My Scala started off much prettier than my Groovy code usually is, but by the time I finished it was an abomination. At that point I gave up and went back to Groovy because while it lacks many of the functional features, idiomatic Groovy "just worked" and gave me most of what I was looking for in terms of a "better Java".

vorg 8y ago
> My Scala started off much prettier than my Groovy code usually is, but by the time I finished it was an abomination > Sometimes I think I should just use Kotlin, but then I would miss the ability to drop seamlessly into dynamic mode Have you considered Clojure for this use case? That might alleviate your problems with using Kotlin and Scala, without having to resort to a single-target language like Apache Groovy. After all, unlike what Clojure, Kotlin, and Scala all offer now, we're never going to see an edition of Groovy that targets Javascript, Android, or native CPU.
- zmmmmm 8y ago
  I've always been intrigued by Clojure but it feels like a huge leap to dive into a fully Lisp-style, completely functional language. I guess single target isn't a particularly big problem for me; the JVM runs everywhere I need it to.

dm3 8y ago

For the data processing tasks my go-to tool is Manifold[1]. Depending on how the application is structured, you can keep state in atoms, the system map when using Component[2] or vars with Mount[3]. Manifold streams asynchronously join the state transformations together while allowing control over backpressure and threading. On the data analysis side things aren't rosy on the JVM. I'd delegate that part to something which actually has all the tools built-in, e.g. Julia/Python+Numpy/R. Apache Arrow[4] is a nice project to facilitate the dataframe interop. [1] https://github.com/ztellman/manifold https://github.com/ztellman/manifold [2] https://github.com/stuartsierra/component https://github.com/stuartsierra/component [3] https://github.com/tolitius/mount https://github.com/tolitius/mount [4] https://arrow.apache.org/ https://arrow.apache.org/