For me, the biggest improvement is the unified typed Dataset API [1]. The current Dataset API gave us a lot of flexibility and type-safety and the new API lets

For me, the biggest improvement is the unified typed Dataset API [1]. The current Dataset API gave us a lot of flexibility and type-safety and the new API lets us use it as DataFrame API instead of converting to RDD and reinventing the wheel, like aggregators [2]. [1] https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6122906529858466/431554386690871/4814681571895601/latest.html https://databricks-prod-cloudfront.cloud.databricks.com/publ... [2] https://docs.cloud.databricks.com/docs/spark/1.6/examples/Dataset%20Aggregator.html https://docs.cloud.databricks.com/docs/spark/1.6/examples/Da...