Would Databricks solve this problem? https://databricks.com/ https://databricks.com/ They are essentially Apache Spark-as-a-service and have an API that allows

Would Databricks solve this problem? https://databricks.com/ https://databricks.com/ They are essentially Apache Spark-as-a-service and have an API that allows you to submit a job on a cluster that you can configure to autoscale: https://community.cloud.databricks.com/doc/api/#jobs.JobsService.createJob https://community.cloud.databricks.com/doc/api/#jobs.JobsSer... https://community.cloud.databricks.com/doc/api/#jobs.ClusterSpec.NewCluster https://community.cloud.databricks.com/doc/api/#jobs.Cluster...