5 ms·

> I really don’t understand the communities obsession with unwieldy tools like DBT. It lets me write test first sql transforms. I never thought TDD sql would b

> I really don’t understand the communities obsession with unwieldy tools like DBT. It lets me write test first sql transforms. I never thought TDD sql would be possible. My sql is so much more readable with common logic extracted into ephmeral models. I practice same method to write clear code to write sql, eg: too many mocks = refactor into separate model ( class) . I think DBT made this possible with refs that can be swapped out with mocks. This is the awesome library I am using https://github.com/EqualExperts/dbt-unit-testing https://github.com/EqualExperts/dbt-unit-testing

akhmatova 4y ago
My sql is so much more readable with common logic extracted into ephmeral models. Sounds great. My concern though with DBT is the cognitive overhead and just plain ... weirdness and clunkiness it seems to introduce as a cost to obtaining benefits like these. For example if we look at their own documentation: When using the view materialization, your model is rebuilt as a view on each run, via a create view as statement. https://docs.getdbt.com/docs/build/materializations https://docs.getdbt.com/docs/build/materializations Now, isn't your reaction, upon reading this, some version of: "Whaaat? Views are not materializations - by definition, they are essentially the opposite of materializations." And yet here the DBT documentation is describing the results of plain old "create view" statements as materializations. And then we're not getting into even more basic issues - like why we're supposed to believe that code like: select ... from {{ source('jaffle_shop', 'orders') }} left join {{ source('jaffle_shop', 'customers') }} using (customer_id) Is more readable or maintainable than what it compiles into: select ... from raw.jaffle_shop.orders left join raw.jaffle_shop.customers using (customer_id) I'm happy it works for you. But I'm just not seeing the magic here.
- mb7733 4y ago
  > Now, isn't your reaction, upon reading this, some version of: "Whaaat? Views are not materializations - by definition, they are essentially the opposite of materializations I'd never heard of dbt before today but from a quick scan of that page it's clear to me what a that 'materialization' is in dbt. It's not the same as a SQL 'materialized view'; it could be a SQL view, an SQL table, a CTE, or a table with extra machinery for incremental updating. In my experience in software it's common for similar words to be used for different concepts in different contexts. I think the dbt documentation is well done so I don't see a big issue.
  - akhmatova 4y ago
    In my experience in software it's common for similar words to be used for different concepts in different contexts. Right, but when it happens it's usually because these words get used differently in different domains. What's happening here is that they're carelessly (in my view) mangling a well-established concept in a field (speaking of the SQL and database world) where folks are (notoriously) careful about word choice and about what things actually, in fact, mean. It just feels weird, and on a certain gut level - koolaid-y.
    - FridgeSeal 4y ago
      The whole hype around dbt and snowflake feels koolaid-y to me. They’re some products that have inexplicably captured so much s as attention for so little value and now trounce around acting like their approach is the only true way of doing things.
FridgeSeal 4y ago
> It lets me write test first sql transforms. I never thought TDD sql would be possible I went through the library you linked, but I’m really, really not convinced. This just looks like a messier, more unclear way to test some SQL. A feat you could with some spark code (for example) in a significantly less confusing and boilerplate-y manner. I think the benefit is that dbt appears to give some cleaner way to separate and the re-compose SQL. Which is good, we do need that. However it does it in the most awkward and unwieldy way possible (string templating) and yet it’s heralded as “the true way forward”.
- mythhouse 4y ago
  Sure a full fledged programming stack will obviously have a more mature tdd facilities. this if you choose to write sql . sql vs spark is probably a discussion for another time :0 > This just looks like a messier, more unclear way to test some SQL. As a software engineer I've written my own share of unit tests. They are supposed to be documentation of how a unit might work but it rarely ever works in that way. I found that to be more true for CSV tests for sql https://github.com/EqualExperts/dbt-unit-testing#different-ways-to-build-mock-values https://github.com/EqualExperts/dbt-unit-testing#different-w... because they are always in the same format , input csv -> output csv I am more motivated to look at the tests to see what sql is doing. Its pretty straight forward.

akhmatova 4y ago

My sql is so much more readable with common logic extracted into ephmeral models. Sounds great. My concern though with DBT is the cognitive overhead and just plain ... weirdness and clunkiness it seems to introduce as a cost to obtaining benefits like these. For example if we look at their own documentation: When using the view materialization, your model is rebuilt as a view on each run, via a create view as statement. https://docs.getdbt.com/docs/build/materializations https://docs.getdbt.com/docs/build/materializations Now, isn't your reaction, upon reading this, some version of: "Whaaat? Views are not materializations - by definition, they are essentially the opposite of materializations." And yet here the DBT documentation is describing the results of plain old "create view" statements as materializations. And then we're not getting into even more basic issues - like why we're supposed to believe that code like: select ... from {{ source('jaffle_shop', 'orders') }} left join {{ source('jaffle_shop', 'customers') }} using (customer_id) Is more readable or maintainable than what it compiles into: select ... from raw.jaffle_shop.orders left join raw.jaffle_shop.customers using (customer_id) I'm happy it works for you. But I'm just not seeing the magic here.

mb7733 4y ago
> Now, isn't your reaction, upon reading this, some version of: "Whaaat? Views are not materializations - by definition, they are essentially the opposite of materializations I'd never heard of dbt before today but from a quick scan of that page it's clear to me what a that 'materialization' is in dbt. It's not the same as a SQL 'materialized view'; it could be a SQL view, an SQL table, a CTE, or a table with extra machinery for incremental updating. In my experience in software it's common for similar words to be used for different concepts in different contexts. I think the dbt documentation is well done so I don't see a big issue.
- akhmatova 4y ago
  In my experience in software it's common for similar words to be used for different concepts in different contexts. Right, but when it happens it's usually because these words get used differently in different domains. What's happening here is that they're carelessly (in my view) mangling a well-established concept in a field (speaking of the SQL and database world) where folks are (notoriously) careful about word choice and about what things actually, in fact, mean. It just feels weird, and on a certain gut level - koolaid-y.
  - FridgeSeal 4y ago
    The whole hype around dbt and snowflake feels koolaid-y to me. They’re some products that have inexplicably captured so much s as attention for so little value and now trounce around acting like their approach is the only true way of doing things.

FridgeSeal 4y ago

> It lets me write test first sql transforms. I never thought TDD sql would be possible I went through the library you linked, but I’m really, really not convinced. This just looks like a messier, more unclear way to test some SQL. A feat you could with some spark code (for example) in a significantly less confusing and boilerplate-y manner. I think the benefit is that dbt appears to give some cleaner way to separate and the re-compose SQL. Which is good, we do need that. However it does it in the most awkward and unwieldy way possible (string templating) and yet it’s heralded as “the true way forward”.

mythhouse 4y ago
Sure a full fledged programming stack will obviously have a more mature tdd facilities. this if you choose to write sql . sql vs spark is probably a discussion for another time :0 > This just looks like a messier, more unclear way to test some SQL. As a software engineer I've written my own share of unit tests. They are supposed to be documentation of how a unit might work but it rarely ever works in that way. I found that to be more true for CSV tests for sql https://github.com/EqualExperts/dbt-unit-testing#different-ways-to-build-mock-values https://github.com/EqualExperts/dbt-unit-testing#different-w... because they are always in the same format , input csv -> output csv I am more motivated to look at the tests to see what sql is doing. Its pretty straight forward.