4 ms·

Yup, it is not perfect but I will take dbt every day against the old way as some random DDL files in git.

CRConrad 3y ago
So basically its huge advantage is that it saves its code as text, which makes it gittable? (Dang, and here the tool I've been designing in my head for several years was going to claim that niche... So unique. ;-)
- camgunz 3y ago
  Nah, the main thing is the dependency resolution. So you make pipelines like: - employees - tall_employees - tall_employees_by_salary These tables (dbt would call them models because they can also be views depending on how they're configured) depend on each other, so you have to build them in a specific order. Without something like this, you're manually running SQL to build your pipelines, and that's very error prone. With something like this you can run tests (they're also SQL in dbt, which is pretty common in data engineering teams), run your pipeline in CI, etc. There's a bunch of other features, but they all hang off this basic idea.
  - riku_iki 3y ago
    > you're manually running SQL to build your pipelines you can have some sh file which runs SQL in desired order..
    - tomnipotent 3y ago
      > runs SQL in desired order Sure, one at a time. dbt builds a DAG of the SQL operations and can run transformations concurrently.
      - riku_iki 3y ago
        and why would you need this? If the goal is to reduce e2e latency, then optimizing SQL is likely much more beneficial, since it can improve performance by NNN times, not just by N times.
        tomnipotent 3y ago
        > and why would you need this? So things finish considerably faster? > then optimizing SQL is likely much more beneficial This only gets you so far. Running transformations one-at-a-time when the server can do more is just holding you back. My last dbt project had over 300+ tables and running transformations serially would have taken an order of magnitude longer than running concurrently.
        riku_iki 3y ago
        > Running transformations one-at-a-time when the server can do more is just holding you back in general, the end goal of such ETL optimization is to have all your CPU or IO saturated. But I imagine for 300 tables transformations it could be too much work.
        tomnipotent 3y ago
        The goal of ETL is to transform data for practical applications, for which CPU and memory is there to be utilized. This particular dbt process was running against the second smallest Redshift instance, which barely noticed when the process kicked off every 10 minutes to run incremental updates. Would run for maybe 60-90 seconds, with only the occasional lock contention with long-running analyst queries that was easily mitigated.
        riku_iki 3y ago
        > The goal of ETL is I said goal of "such ETL optimization", not ETL. Luckily for me it is an hour and 50 lines(likely less) of code to write such parallel executor with dependency tracking if need will arise.
- benjaminwootton 3y ago
  This is an old sub-thread now, but I would agree that dbt is a very simple idea and easy to use at its core. Store your models in text files, in git, add a dag. It’s easy to use and very lightweight so I don’t see any reason to roll your own. Then you get things like testing, documentation, modularity if you want it.

So basically its huge advantage is that it saves its code as text, which makes it gittable? (Dang, and here the tool I've been designing in my head for several years was going to claim that niche... So unique. ;-)

camgunz 3y ago
Nah, the main thing is the dependency resolution. So you make pipelines like: - employees - tall_employees - tall_employees_by_salary These tables (dbt would call them models because they can also be views depending on how they're configured) depend on each other, so you have to build them in a specific order. Without something like this, you're manually running SQL to build your pipelines, and that's very error prone. With something like this you can run tests (they're also SQL in dbt, which is pretty common in data engineering teams), run your pipeline in CI, etc. There's a bunch of other features, but they all hang off this basic idea.
- riku_iki 3y ago
  > you're manually running SQL to build your pipelines you can have some sh file which runs SQL in desired order..
  - tomnipotent 3y ago
    > runs SQL in desired order Sure, one at a time. dbt builds a DAG of the SQL operations and can run transformations concurrently.
    - riku_iki 3y ago
      and why would you need this? If the goal is to reduce e2e latency, then optimizing SQL is likely much more beneficial, since it can improve performance by NNN times, not just by N times.
      - tomnipotent 3y ago
        > and why would you need this? So things finish considerably faster? > then optimizing SQL is likely much more beneficial This only gets you so far. Running transformations one-at-a-time when the server can do more is just holding you back. My last dbt project had over 300+ tables and running transformations serially would have taken an order of magnitude longer than running concurrently.
        riku_iki 3y ago
        > Running transformations one-at-a-time when the server can do more is just holding you back in general, the end goal of such ETL optimization is to have all your CPU or IO saturated. But I imagine for 300 tables transformations it could be too much work.
        tomnipotent 3y ago
        The goal of ETL is to transform data for practical applications, for which CPU and memory is there to be utilized. This particular dbt process was running against the second smallest Redshift instance, which barely noticed when the process kicked off every 10 minutes to run incremental updates. Would run for maybe 60-90 seconds, with only the occasional lock contention with long-running analyst queries that was easily mitigated.
        riku_iki 3y ago
        > The goal of ETL is I said goal of "such ETL optimization", not ETL. Luckily for me it is an hour and 50 lines(likely less) of code to write such parallel executor with dependency tracking if need will arise.
benjaminwootton 3y ago
This is an old sub-thread now, but I would agree that dbt is a very simple idea and easy to use at its core. Store your models in text files, in git, add a dag. It’s easy to use and very lightweight so I don’t see any reason to roll your own. Then you get things like testing, documentation, modularity if you want it.