3 ms·

Hi, Thanks for the answer. What you said resonates with me - with a few changes. Spark 2.3 will come with Arrow UDF, that should be a significant performance bo

Hi, Thanks for the answer. What you said resonates with me - with a few changes. Spark 2.3 will come with Arrow UDF, that should be a significant performance boost. In that way, yes - we are taking at a forward looking bet. About mllib - yes, we concur with you on algorithmic coverage. And yes, training is the major issue. For example, what I read of Uber's Michaelangelo infrastructure - it seems they train using Spark and save to a custom format that is deserialized (using custom code) and made available as a docker image . There is value in consistency - using Spark thtoy2and through. Wonder what you thought of that ?

wenc 9y ago
1) I've heard about vectorized Python UDFs in Spark 2.3. Thanks for reminding of that. https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html https://databricks.com/blog/2017/10/30/introducing-vectorize... 2) I'm not that familiar with what Uber is doing. My take is I'd like to use Spark for as much as I can, but there are parts that are either more performant or easier to accomplish in Python. Spark with Arrow will definitely change the game.