4 ms·
AWS Redshift vs. Parquets and Apache Spark
I've been mulling this over, and I can't make up my mind. What's better to use for a large data warehouse, AWS Redshift vs Apache Spark on top of Parquet files stored in s3? There really doesn't seem to be much different to me besides you can probably do "more" transforms with Spark. But the complexity is probably reduced with Redshift. I need thoughts and input.
- rahulj51 6y agoCost will definitely be a factor but if it's not then I'd suggest Redshift because of the simplicity. Another aspect is how the data will be consumed - Analysts are typically more comfortable using SQL + Redshift. I'd avoid Spark except for very specific use cases as the operational toil is higher with Spark.
- dancrystalbeach 6y agothe consumption is very SQLish based. "operational toil is higher with Spark" <- I agree with this, this is why I'm leaning towards Redshift. Throwing away half the codebase in favor of some SQL statements is tempting.
- cmollis 6y agohow about the difference in cost? Redshift is pretty expensive. We use Spark to convert our data to Parquet and Presto to query it (although you can just use Spark to query it also)
- dancrystalbeach 6y agoI've thought about that, i should probably do more detailed analysis of what the actual cost difference would be. I've looked at Presto as well... but again that is just another tech stack to manage.
- PeterZaitsev 6y agoIf you want cost efficient high performance Clickhouse is hard to beat
- emrah 6y agoSnowflake :) Redshift is also good See this for comparison benchmarking: https://fivetran.com/blog/warehouse-benchmark https://fivetran.com/blog/warehouse-benchmark