4 ms·

One thing that totally surprised me is that they claim to support Spark, but only if the Spark application runs in the Pinot cluster. There is no Pinot spark wr

One thing that totally surprised me is that they claim to support Spark, but only if the Spark application runs in the Pinot cluster. There is no Pinot spark writer format, you need to change the way you’re deploying Spark drastically for example if you’re using Databricks or K8s as a master.

cpsoman 3y ago
There is indeed Spark support for writing new data into Pinot (https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark https://docs.pinot.apache.org/basics/data-import/batch-inges...) as well as to query it (https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-spark-3-connector/README.md https://github.com/apache/pinot/blob/master/pinot-connectors...). This does not run inside the Pinot cluster - you can use standard Spark execution engine to run this ingestion. In addition, Pinot also supports an out of the box ingestion capability from batch sources using the Minion framework (https://docs.pinot.apache.org/basics/components/cluster/minion#built-in-tasks https://docs.pinot.apache.org/basics/components/cluster/mini...) that does not need any external component (like Spark)
- Lucasoato 3y ago
  This is actually very different from all the rest of Spark integrations publicly available. Remember that while the setups in the docs you linked were normal 10 years ago (I'm thinking about old and expensive cloudera clusters), right now most of the people can use Spark without ever needing to use the spark-submit CLI (ex. Databricks). The way proposed forces the user to separate Spark applications, what if you want to write the result of a Dataframe and save it directly to Pinot? Most of other storage service support it (jdbc, S3, redshift ... ) This is not just a flavor preference, this just wouldn't work with the evolving Spark ecosystem (like Spark Connect or even just PySpark).

cpsoman 3y ago

There is indeed Spark support for writing new data into Pinot (https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark https://docs.pinot.apache.org/basics/data-import/batch-inges...) as well as to query it (https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-spark-3-connector/README.md https://github.com/apache/pinot/blob/master/pinot-connectors...). This does not run inside the Pinot cluster - you can use standard Spark execution engine to run this ingestion. In addition, Pinot also supports an out of the box ingestion capability from batch sources using the Minion framework (https://docs.pinot.apache.org/basics/components/cluster/minion#built-in-tasks https://docs.pinot.apache.org/basics/components/cluster/mini...) that does not need any external component (like Spark)

Lucasoato 3y ago
This is actually very different from all the rest of Spark integrations publicly available. Remember that while the setups in the docs you linked were normal 10 years ago (I'm thinking about old and expensive cloudera clusters), right now most of the people can use Spark without ever needing to use the spark-submit CLI (ex. Databricks). The way proposed forces the user to separate Spark applications, what if you want to write the result of a Dataframe and save it directly to Pinot? Most of other storage service support it (jdbc, S3, redshift ... ) This is not just a flavor preference, this just wouldn't work with the evolving Spark ecosystem (like Spark Connect or even just PySpark).