4 ms·
I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via
by MrPowers 5y ago
I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function:
def input_file_name():
sc = SparkContext._active_spark_context
return Column(sc._jvm.functions.input_file_name())
You can see it's just using py4j to invoke a JVM function.
- marcinzm 5y agoYou can use PySpark without ever writing or thinking about Java code. The library handles all the interop for you and even does vectorized serialization with PyArrow. The same way machine learning code is actually mostly C++ or CUDA but you almost never think of that when using Python libraries.