This is actually very different from all the rest of Spark integrations publicly available. Remember that while the setups in the docs you linked were normal 1

This is actually very different from all the rest of Spark integrations publicly available. Remember that while the setups in the docs you linked were normal 10 years ago (I'm thinking about old and expensive cloudera clusters), right now most of the people can use Spark without ever needing to use the spark-submit CLI (ex. Databricks). The way proposed forces the user to separate Spark applications, what if you want to write the result of a Dataframe and save it directly to Pinot? Most of other storage service support it (jdbc, S3, redshift ... ) This is not just a flavor preference, this just wouldn't work with the evolving Spark ecosystem (like Spark Connect or even just PySpark).