4 ms·
What does Clojure bring? Here is my data science stack https://github.com/mratsim/arch-Data-Science https://github.com/mratsim/arch-Data-Science Let's go over
by mratsim 6y ago
What does Clojure bring?
Here is my data science stack https://github.com/mratsim/arch-Data-Science https://github.com/mratsim/arch-Data-Science
Let's go over it:
- Is there a mature tensor/multidimensional array library in Clojure? Is there a Dataframe library?
- Are they compatible with common formats: .npy, .csv, .hdf5 and sqlite databases
- Is there a fast PCA and DBSCAN algorithm?
- Can you pass those arrays to XGBoost or LightGBM for Gradient Boosting based classification or regression.
- Do they support out-of-core processing (i.e. memory-mapped arrays) for processing that takes more than your RAM (happens often with geolocalisation)
- Can you use PyTorch through Clojure? Can you use Tensorflow? What's the GPU computing story
- Does your ML library support things like One-Hot-Encoding, cross-validation? Or more advanced like Target Encoding?
- NLP: stanfordNLP wrapper? spacy? Huggingfaces? Rasa-nlu? sentencepieces?
- Deep learning: Do you support image augmentation like Nvidia DALi or Albumentations? Mixed precision training like Nvidia APEx
- Vision: How is the OpenCV support? Does it interface with Clojure multidimensional arrays? Do you have morphological operations like erosion or dilation?
- Visualization: is there a dataviz library like plotly, ggplot2 or Altair/Vega?
And more importantly
- Are there tutorials and courses in Clojure from beginner to advanced concept? Even Julia is struggling with that.
- mnemonicsloth 6y agoIn other words, you'd consider switching to Java if it was exactly like the system you already use. And note the question you didn't ask: does Java do anything better than python? Didn't even cross your mind.
- dunefox 6y agoHe wouldn't because Java gets in your way much more than Python does. A better language needs to offer these things AND be a better language.
- mnemonicsloth 6y agoAgreed that Java is an obnoxious language. I think originally the grandparent and I were talking about Clojure, which uses the Java library ecosystem, and I started talking about Java libs. Clojure is certainly the context I'd use Java's machine learning libraries in.
- AnimalMuppet 6y agoWell then, instead of putting words in his mouth like "exactly the same", go down his list. > - Is there a mature tensor/multidimensional array library in Clojure? Is there a Dataframe library? Is there one in Java? If so, what? And so on with his other items. It doesn't have to be exactly the same (offering him the benefit of the doubt), but it has to offer substantially the same functionality with substantially the same performance. So, you want to use Java libraries from Clojure? Go for it. Where are the things he needs?
- pjmlp 6y agoNot sure what they cover, but there are surely libraries available, https://deeplearning4j.org/ https://deeplearning4j.org/ https://github.com/originrose/cortex https://github.com/originrose/cortex http://www.onyxplatform.org/ http://www.onyxplatform.org/ https://neanderthal.uncomplicate.org/ https://neanderthal.uncomplicate.org/ https://clojurecuda.uncomplicate.org/ https://clojurecuda.uncomplicate.org/ https://clojurecl.uncomplicate.org/ https://clojurecl.uncomplicate.org/ https://github.com/uncomplicate/bayadera https://github.com/uncomplicate/bayadera https://devblogs.nvidia.com/grcuda-a-polyglot-language-binding-for-cuda-in-graalvm/ https://devblogs.nvidia.com/grcuda-a-polyglot-language-bindi... Also since Pytorch 1.4 there is initial support for Java, built on top of the new pure C++ API.
- mratsim 6y agoThe ceremony to start using Java is significantly worse than Python or Clojure. Science is about functions, not objects, a language that requires all functions to be associated with an object is a non-starter.