4 ms·
Why would Spark trounce Snowflake. What makes it inherently so much faster at 100TB jobs? Also what kind of queries are we talking about?
by nojvek 5y ago
Why would Spark trounce Snowflake. What makes it inherently so much faster at 100TB jobs?
Also what kind of queries are we talking about?
- saj1th 5y ago> Why would Spark trounce Snowflake. What makes it inherently so much faster at 100TB jobs? These are the slides from a talk one of the co-founders (@rxin) gave at Stanford. https://web.stanford.edu/class/cs245/slides/LakehouseGuestTalk.pdf https://web.stanford.edu/class/cs245/slides/LakehouseGuestTa... It goes into the details of how this performance is achieved(and not just at 100TB). Part of this could be attributed to innovations in the storage layer(delta lake), and part of it is just the new query engine design itself.