4 ms·

There is indeed Spark support for writing new data into Pinot (https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark https://docs.pinot.apache.

There is indeed Spark support for writing new data into Pinot (https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark https://docs.pinot.apache.org/basics/data-import/batch-inges...) as well as to query it (https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-spark-3-connector/README.md https://github.com/apache/pinot/blob/master/pinot-connectors...). This does not run inside the Pinot cluster - you can use standard Spark execution engine to run this ingestion. In addition, Pinot also supports an out of the box ingestion capability from batch sources using the Minion framework (https://docs.pinot.apache.org/basics/components/cluster/minion#built-in-tasks https://docs.pinot.apache.org/basics/components/cluster/mini...) that does not need any external component (like Spark)

Lucasoato 3y ago
This is actually very different from all the rest of Spark integrations publicly available. Remember that while the setups in the docs you linked were normal 10 years ago (I'm thinking about old and expensive cloudera clusters), right now most of the people can use Spark without ever needing to use the spark-submit CLI (ex. Databricks). The way proposed forces the user to separate Spark applications, what if you want to write the result of a Dataframe and save it directly to Pinot? Most of other storage service support it (jdbc, S3, redshift ... ) This is not just a flavor preference, this just wouldn't work with the evolving Spark ecosystem (like Spark Connect or even just PySpark).