4 ms·

As someone who spends a lot of time in this space, their only "killer app" is automated workload / data distribution management. Which is cool, and hard to get

As someone who spends a lot of time in this space, their only "killer app" is automated workload / data distribution management. Which is cool, and hard to get right, but clearly something the cloud vendors and other data players are have taken steps towards / offer more or less the same outcomes. And in contrast their Silicon Valley roots means a lot of their tooling/UX/data capabilities are ... undercooked. Their Web IDE feels like a throwback to 2003 Hadoop, their ETL capabilities are a joke, they don't support joins in views ... And they've also squandered some opportunities to actually offer a differentiating "all in one" data processing experience for ad hoc/exploratory, BI/aggregated, and Big Data/AI/ML model crunching. For example, here's their garbage blog post on Spark SQL - https://www.snowflake.com/blog/snowflake-spark-part-2-pushing-query-processing/ https://www.snowflake.com/blog/snowflake-spark-part-2-pushin... tl;dr when someone writes a Spark job that includes a filter against data in Snowflake, it's more efficient to let Snowflake filter the data before shipping it off to the (much more performant) Spark engine to do the actual analytical pieces of the query plan, instead of just shipping all the data over and letting Spark do the filtering. Like ... wow, predicate pushdown is your answer? Contrast with Azure Synapse providing Spark and SQL Server compute in the same environment; Databricks adding Delta Lake capabilities to be more schema-on-write friendly; Dremio building AI into their caching, and Starburst into their workload management ... Anyway, I don't see any secret sauce, which means it's still just traditional enterprise sales cycles...

chrisjc 6y ago
> they don't support joins in views Perhaps you mean they don't support joins in Materialized Views (uppercase M)? We use Snowflake views with joins all over the place. Furthermore if views don't cut it for you, you can always use joins in UDTFs. Or if you really need joins in a materialize view (lowercase M), you can use change streams in combination with joins to maintain your own materialized view (table). > instead of just shipping all the data over and letting Spark do the filtering Forgive my ignorance, but in what capacity would this be less efficient? Doesn't it make more sense to reduce you result set before shipping it off to external compute?
- kthejoker2 6y ago
  To the first point, yes, in materialized views, thanks for correcting me. I perceive it as a limitation of their micropartitioning working against the actual goals of such a system, i.e. pushing even more work on data engineers to prep data. And we're in agreement on the second part: my point is not that their solution isn't more efficient - it is - but it's treating predicate pushdown as some kind of deep synergy between Spark and Snowflake. That is, it's more the marketing aspect - clueless execs see "Snowflake and Spark work great together" and a box is checked.
AznHisoka 6y ago
Are you familiar with Alteryx and if so, do you see Snowflake eventually replacing them?
- macklemoreshair 6y ago
  alteryx will continue to exist as long as people use spreadsheets.

chrisjc 6y ago

> they don't support joins in views Perhaps you mean they don't support joins in Materialized Views (uppercase M)? We use Snowflake views with joins all over the place. Furthermore if views don't cut it for you, you can always use joins in UDTFs. Or if you really need joins in a materialize view (lowercase M), you can use change streams in combination with joins to maintain your own materialized view (table). > instead of just shipping all the data over and letting Spark do the filtering Forgive my ignorance, but in what capacity would this be less efficient? Doesn't it make more sense to reduce you result set before shipping it off to external compute?

kthejoker2 6y ago
To the first point, yes, in materialized views, thanks for correcting me. I perceive it as a limitation of their micropartitioning working against the actual goals of such a system, i.e. pushing even more work on data engineers to prep data. And we're in agreement on the second part: my point is not that their solution isn't more efficient - it is - but it's treating predicate pushdown as some kind of deep synergy between Spark and Snowflake. That is, it's more the marketing aspect - clueless execs see "Snowflake and Spark work great together" and a box is checked.

AznHisoka 6y ago

Are you familiar with Alteryx and if so, do you see Snowflake eventually replacing them?

macklemoreshair 6y ago
alteryx will continue to exist as long as people use spreadsheets.