It is however the push recently in Spark seems to be for better/faster Python support more than anything else. Probably because Databricks makes it's money from

It is however the push recently in Spark seems to be for better/faster Python support more than anything else. Probably because Databricks makes it's money from it's hosted notebooks environment which caters more towards non-engineers (who in turn would be more put off by having to learn Scala I'm guessing).