3 ms·
> they don't support joins in views Perhaps you mean they don't support joins in Materialized Views (uppercase M)? We use Snowflake views with joins all over t
by chrisjc 6y ago
> they don't support joins in views
Perhaps you mean they don't support joins in Materialized Views (uppercase M)? We use Snowflake views with joins all over the place. Furthermore if views don't cut it for you, you can always use joins in UDTFs. Or if you really need joins in a materialize view (lowercase M), you can use change streams in combination with joins to maintain your own materialized view (table).
> instead of just shipping all the data over and letting Spark do the filtering
Forgive my ignorance, but in what capacity would this be less efficient? Doesn't it make more sense to reduce you result set before shipping it off to external compute?
- kthejoker2 6y agoTo the first point, yes, in materialized views, thanks for correcting me. I perceive it as a limitation of their micropartitioning working against the actual goals of such a system, i.e. pushing even more work on data engineers to prep data. And we're in agreement on the second part: my point is not that their solution isn't more efficient - it is - but it's treating predicate pushdown as some kind of deep synergy between Spark and Snowflake. That is, it's more the marketing aspect - clueless execs see "Snowflake and Spark work great together" and a box is checked.