3 ms·
Spark has always been able to handle way larger scale than any DW.
by feqgmmr2 5y ago
Spark has always been able to handle way larger scale than any DW.
- hello_moto 5y agoHandle what though? Can Spark queries 100Bn structured data performing aggregation on multiple fields (or dimension?)
- geoduck14 5y agoIn my previous company, we had 63 petabytes of data in Snowflake.
- hello_moto 5y agoThat sounds great: storage problem is solved. What about large scale read via OLAP queries (y'know, the typical measures and dimensions)
- blueglassfish 5y agoThat's a respectable amount for a DW, true. Spark and it's ilk are designed for much larger scales though. Multiple FAANG use cases for Spark are in the petabytes per week range.