2 ms·

We’ve recently introduced dbt, fivetran and snowflake and it has been amazing on many levels. Speed of development along with CI/CD has been fantastic. Clarity

We’ve recently introduced dbt, fivetran and snowflake and it has been amazing on many levels. Speed of development along with CI/CD has been fantastic. Clarity of thought about organizing our work has helped with the more junior and platform specific people. Before I got into it I was sceptical because I was thinking It’s just SQL and SQL is pretty easy and straightforward. It breaks bad habits in DW development. I’m not sure I still get why but it has been one of the few things Ive seen in a long time in this area that has been a big jump in capabilities. There is so much more confidence on my team. We are moving much faster. We can just get on with delivering without having to undo and think of everything is precious or dangerous to change. It took some hands on work and getting over the initial mind bend to get there but I wouldn’t go back to what we had. I would likely only make my next move if they had a similar setup in place or were open to change.

thejosh 5y ago
Exactly what we've been doing for 2+ years at my job. It's amazing. Snowflake is amazing, but watch out for search optimization costs (it's great for append only), left joins taking FOREVER (avoid left joins as much as possible for large datasets).
- theplague42 5y ago
  What makes left joins perform poorly in Snowflake?
  - thejosh 5y ago
    It has to join each part of the previous join to the next join, and if you have a lot of joins this can get out of hand. We have a lot of joins in our final fct orders from our intermediate table, and looks like this: from foo left join bar on bar.common_id = foo.common_id left join baz on baz.common_id = foo.common_id left join qux on qux.common_id = foo.common_id left join waldo on waldo.common_id = foo.common_id So waldo joins to qux, which joins to qux... I call it a "staircase join", as that's what it looks like in the SF profiler.

thejosh 5y ago

Exactly what we've been doing for 2+ years at my job. It's amazing. Snowflake is amazing, but watch out for search optimization costs (it's great for append only), left joins taking FOREVER (avoid left joins as much as possible for large datasets).

theplague42 5y ago
What makes left joins perform poorly in Snowflake?
- thejosh 5y ago
  It has to join each part of the previous join to the next join, and if you have a lot of joins this can get out of hand. We have a lot of joins in our final fct orders from our intermediate table, and looks like this: from foo left join bar on bar.common_id = foo.common_id left join baz on baz.common_id = foo.common_id left join qux on qux.common_id = foo.common_id left join waldo on waldo.common_id = foo.common_id So waldo joins to qux, which joins to qux... I call it a "staircase join", as that's what it looks like in the SF profiler.