5 ms·
You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes
by groceryheist 6y ago
You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.