3 ms·
Karthik, I'm no Spark expert but almost all advice I read is to avoid UDFs if at all possible. Examples below: - https://medium.com/teads-engineering/spark-per
by maslam 8y ago
Karthik, I'm no Spark expert but almost all advice I read is to avoid UDFs if at all possible. Examples below:
- https://medium.com/teads-engineering/spark-performance-tuning-from-the-trenches-7cbde521cf60 https://medium.com/teads-engineering/spark-performance-tunin...
- https://www.inovex.de/blog/efficient-udafs-with-pyspark/ https://www.inovex.de/blog/efficient-udafs-with-pyspark/
- karthiksr 8y agoThank you for those pointers. There are definitely some differences between the kind of UDFs that Spark supports and the kind that Froid handles. For one, Spark UDFs cannot invoke a Spark SQL query in their definition AFAIK, whereas TSQL functions can. But still, some techniques might be applicable. Definitely worth digging further!