4 ms·

Hopefully this can pave the way for deep learning and data analysis for Scala/Java. It's really frustrating having to use a subpar and inexpressive language lik

by mruts 7y ago

Hopefully this can pave the way for deep learning and data analysis for Scala/Java. It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect. Or if Julia could get more libraries (it's maybe 25% there, which isn't bad), that would be pretty great as well.

vturner 7y ago
Not to mention it's a nightmare to scale...
Something1234 7y ago
How is java expressive when you don't have operator overloads?
- brabel 7y ago
  Java libraries can easily be used from any JVM language, including many that have operator overloads and much more (Scala, Kotlin, Groovy, Eta etc.)
rand84545 7y ago
> inexpressive language like Python? Could you elaborate on this please?
- mruts 7y ago
  Crippled lambdas, no pattern matching, no static typing (also no HKT), no macros, no threads, no actors, no currying, no practical FP, the list goes on and on.. Moreover, the performance is awful, everything is boxed, and there’s no support for tail call eliminating. Compared to Scala, OCaml, Common Lisp, or Racket, Python doesn’t even come close in terms of power and speed as the aforementioned languages. I use python everyday and I find it very hard to work with. I do quant/ml/data science work with it and python has the best libraries of any language in those fields. But there’s no real reason why that should be the case besides maybe that python is so lacking in features that it takes 30 minutes to learn.
deleted 7y ago
[deleted]
geoalchimista 7y ago
> It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect. I'm afraid that the "network effect" is gonna continue for the years to come. NDArray is not the only thing you need for data analysis. DataFrame, statistical functions, and data visualization packages like matplotlib and ggplot are all indispensable components in a data analysis workflow. It has taken thousands of contributors a decade to build a solid ecosystem for data science in Python or in R. It's just unlikely for Java to become a mainstream data science language overnight. I'd say if you are looking for a statically typed language for deep learning, C++ is probably a better bet than Java for now.
- stochastic_monk 7y ago
  PyTorch and Tensorflow both have C++ frontends now, which lowers the barrier of use for me. Being able to test quickly in Python without compiling is really nice; while I prefer C++ for final products, prototyping PyTorch in Python feels like a better use of my time. There’s a fuller ecosystem; I needed a PCA implementation. Instead of just using scipy, I had to choose an algorithm and use LAPACK myself.
- mruts 7y ago
  I don’t really want Java as a data science language, but Scala would be a perfect fit. There’s already a nice data frame library that I helped develop at my old company called Saddle. Now we just need some quant finance libraries (like the excellent zip line, pyfolio, and alpha lens), some industrial NNs, optimization libs (like cvxpy), and something like scipy. Is this ever going to happen? Probably not, but a man can dream..
specialist 7y ago
I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits. Madness. I am still aghast. How does anyone get any work done using those tools? https://www.datacamp.com/community/tutorials/apache-spark-python#difference https://www.datacamp.com/community/tutorials/apache-spark-py... Why didn't they implement RowSet interface? Or create new tabular data holding interface? Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane. -- The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?
- mruts 7y ago
  Spark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.
  - stochastic_monk 7y ago
    Have you looked at Thrill? [0] It claims to basically be a better, C++ Spark. [0] https://github.com/thrill https://github.com/thrill

by mruts 7y ago

vturner 7y ago

Not to mention it's a nightmare to scale...

Something1234 7y ago

How is java expressive when you don't have operator overloads?

brabel 7y ago
Java libraries can easily be used from any JVM language, including many that have operator overloads and much more (Scala, Kotlin, Groovy, Eta etc.)

rand84545 7y ago

> inexpressive language like Python? Could you elaborate on this please?

mruts 7y ago
Crippled lambdas, no pattern matching, no static typing (also no HKT), no macros, no threads, no actors, no currying, no practical FP, the list goes on and on.. Moreover, the performance is awful, everything is boxed, and there’s no support for tail call eliminating. Compared to Scala, OCaml, Common Lisp, or Racket, Python doesn’t even come close in terms of power and speed as the aforementioned languages. I use python everyday and I find it very hard to work with. I do quant/ml/data science work with it and python has the best libraries of any language in those fields. But there’s no real reason why that should be the case besides maybe that python is so lacking in features that it takes 30 minutes to learn.

deleted 7y ago

[deleted]

geoalchimista 7y ago

> It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect. I'm afraid that the "network effect" is gonna continue for the years to come. NDArray is not the only thing you need for data analysis. DataFrame, statistical functions, and data visualization packages like matplotlib and ggplot are all indispensable components in a data analysis workflow. It has taken thousands of contributors a decade to build a solid ecosystem for data science in Python or in R. It's just unlikely for Java to become a mainstream data science language overnight. I'd say if you are looking for a statically typed language for deep learning, C++ is probably a better bet than Java for now.

stochastic_monk 7y ago
PyTorch and Tensorflow both have C++ frontends now, which lowers the barrier of use for me. Being able to test quickly in Python without compiling is really nice; while I prefer C++ for final products, prototyping PyTorch in Python feels like a better use of my time. There’s a fuller ecosystem; I needed a PCA implementation. Instead of just using scipy, I had to choose an algorithm and use LAPACK myself.
mruts 7y ago
I don’t really want Java as a data science language, but Scala would be a perfect fit. There’s already a nice data frame library that I helped develop at my old company called Saddle. Now we just need some quant finance libraries (like the excellent zip line, pyfolio, and alpha lens), some industrial NNs, optimization libs (like cvxpy), and something like scipy. Is this ever going to happen? Probably not, but a man can dream..

specialist 7y ago

I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits. Madness. I am still aghast. How does anyone get any work done using those tools? https://www.datacamp.com/community/tutorials/apache-spark-python#difference https://www.datacamp.com/community/tutorials/apache-spark-py... Why didn't they implement RowSet interface? Or create new tabular data holding interface? Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane. -- The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?

mruts 7y ago
Spark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.
- stochastic_monk 7y ago
  Have you looked at Thrill? [0] It claims to basically be a better, C++ Spark. [0] https://github.com/thrill https://github.com/thrill