1) I've heard about vectorized Python UDFs in Spark 2.3. Thanks for reminding of that. https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-p

1) I've heard about vectorized Python UDFs in Spark 2.3. Thanks for reminding of that. https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html https://databricks.com/blog/2017/10/30/introducing-vectorize... 2) I'm not that familiar with what Uber is doing. My take is I'd like to use Spark for as much as I can, but there are parts that are either more performant or easier to accomplish in Python. Spark with Arrow will definitely change the game.