3 ms·

DBT is really nice, but not a game changer since it's basically a SQL generator and only as good as the underlying SQL engine of your data warehouse.

by dikei 5y ago

beckingz 5y ago
The beautiful (and scary thing) about cloud data warehouses is that you can scale them significantly, which means bad SQL mostly becomes a cost issue as opposed to a speed issue. Also, dbt makes it easier to persist tables instead of views, which has a massive performance improvement.
ziga 5y ago
The dbt community will frequently run Snowflake, and its SQL engine is very good. https://medium.com/@jthandy/how-compatible-are-redshift-and-snowflakes-sql-syntaxes-c2103a43ae84 https://medium.com/@jthandy/how-compatible-are-redshift-and-...
lunatuna 5y ago
We’ve recently introduced dbt, fivetran and snowflake and it has been amazing on many levels. Speed of development along with CI/CD has been fantastic. Clarity of thought about organizing our work has helped with the more junior and platform specific people. Before I got into it I was sceptical because I was thinking It’s just SQL and SQL is pretty easy and straightforward. It breaks bad habits in DW development. I’m not sure I still get why but it has been one of the few things Ive seen in a long time in this area that has been a big jump in capabilities. There is so much more confidence on my team. We are moving much faster. We can just get on with delivering without having to undo and think of everything is precious or dangerous to change. It took some hands on work and getting over the initial mind bend to get there but I wouldn’t go back to what we had. I would likely only make my next move if they had a similar setup in place or were open to change.
- thejosh 5y ago
  Exactly what we've been doing for 2+ years at my job. It's amazing. Snowflake is amazing, but watch out for search optimization costs (it's great for append only), left joins taking FOREVER (avoid left joins as much as possible for large datasets).
  - theplague42 5y ago
    What makes left joins perform poorly in Snowflake?
    - thejosh 5y ago
      It has to join each part of the previous join to the next join, and if you have a lot of joins this can get out of hand. We have a lot of joins in our final fct orders from our intermediate table, and looks like this: from foo left join bar on bar.common_id = foo.common_id left join baz on baz.common_id = foo.common_id left join qux on qux.common_id = foo.common_id left join waldo on waldo.common_id = foo.common_id So waldo joins to qux, which joins to qux... I call it a "staircase join", as that's what it looks like in the SF profiler.

DBT is really nice, but not a game changer since it's basically a SQL generator and only as good as the underlying SQL engine of your data warehouse.

by dikei 5y ago

DBT is really nice, but not a game changer since it's basically a SQL generator and only as good as the underlying SQL engine of your data warehouse.

beckingz 5y ago

The beautiful (and scary thing) about cloud data warehouses is that you can scale them significantly, which means bad SQL mostly becomes a cost issue as opposed to a speed issue. Also, dbt makes it easier to persist tables instead of views, which has a massive performance improvement.

ziga 5y ago

The dbt community will frequently run Snowflake, and its SQL engine is very good. https://medium.com/@jthandy/how-compatible-are-redshift-and-snowflakes-sql-syntaxes-c2103a43ae84 https://medium.com/@jthandy/how-compatible-are-redshift-and-...

lunatuna 5y ago

We’ve recently introduced dbt, fivetran and snowflake and it has been amazing on many levels. Speed of development along with CI/CD has been fantastic. Clarity of thought about organizing our work has helped with the more junior and platform specific people. Before I got into it I was sceptical because I was thinking It’s just SQL and SQL is pretty easy and straightforward. It breaks bad habits in DW development. I’m not sure I still get why but it has been one of the few things Ive seen in a long time in this area that has been a big jump in capabilities. There is so much more confidence on my team. We are moving much faster. We can just get on with delivering without having to undo and think of everything is precious or dangerous to change. It took some hands on work and getting over the initial mind bend to get there but I wouldn’t go back to what we had. I would likely only make my next move if they had a similar setup in place or were open to change.

thejosh 5y ago
Exactly what we've been doing for 2+ years at my job. It's amazing. Snowflake is amazing, but watch out for search optimization costs (it's great for append only), left joins taking FOREVER (avoid left joins as much as possible for large datasets).
- theplague42 5y ago
  What makes left joins perform poorly in Snowflake?
  - thejosh 5y ago
    It has to join each part of the previous join to the next join, and if you have a lot of joins this can get out of hand. We have a lot of joins in our final fct orders from our intermediate table, and looks like this: from foo left join bar on bar.common_id = foo.common_id left join baz on baz.common_id = foo.common_id left join qux on qux.common_id = foo.common_id left join waldo on waldo.common_id = foo.common_id So waldo joins to qux, which joins to qux... I call it a "staircase join", as that's what it looks like in the SF profiler.