6 ms·
Did you evaluate Julia (or already have experience with it)?
by leetrout 5y ago
Did you evaluate Julia (or already have experience with it)?
- marcinzm 5y agoThe issue is that 95% of the ML ecosystem is in Python so if you're doing ML your best options are Python and things that integrated with Python (ie: PySpark).
- MrPowers 5y agoI agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function: def input_file_name(): sc = SparkContext._active_spark_context return Column(sc._jvm.functions.input_file_name()) You can see it's just using py4j to invoke a JVM function.
- marcinzm 5y agoYou can use PySpark without ever writing or thinking about Java code. The library handles all the interop for you and even does vectorized serialization with PyArrow. The same way machine learning code is actually mostly C++ or CUDA but you almost never think of that when using Python libraries.
- MrPowers 5y agoI haven't tried Julia yet, but suspect there is a large class of problems that are best suited for Julia. Your question motivated me to do a bit a research and it looks like a cloud based Julia runtime exists (JuliaHub). It looks like a promising solution, especially if you need to leverage a Julia lib.