3 ms·

Well there's 2 main reasons. The first is that you can update a subset of your pipeline, like say you have an internal dashboard for server stats and another e

Well there's 2 main reasons. The first is that you can update a subset of your pipeline, like say you have an internal dashboard for server stats and another external dashboard that's a product for customers, and you notice a problem w/ the server stats dashboard. You can have dbt just build the models that underlie the server stats dashboard and also exempt any model used by the external dashboard. The second is that you can effectively defer processing. dbt builds views by default (though it of course will also build tables, including ephemeral tables that are cleaned up after a build), which means your transforms don't take place until you query the data. If you can get away with it, this is what you want, and there are lots of ways to stay on views (caching, use BigQuery, etc). This means your pipeline at least has the potential of being "real time". But, overall we're talking about pipelines that have 100s or 1000s of models. You need some kind of system for that; a shell script isn't gonna cut it.

riku_iki 3y ago
You obviously can have modularity with shell scripts: run_all_etl.sh build_internal_dashboard_etl.sh (called from run_all_etl.sh) build_external_dashboard_etl.sh (called from run_all_etl.sh)
- itsoktocry 3y ago
  How many models are in "run all etl", and what if one fails? You can't see how issues can arise here, with what you describe? I mean, go ahead and write your SQL pipeline in bash. Many of us are using dbt because we see a lot of benefit, we aren't all morons.
  - riku_iki 3y ago
    > How many models are in "run all etl", and what if one fails? hundred tables. one fail -> whole pipeline fails to ensure e2e and overall consistency.
- camgunz 3y ago
  No you misunderstand. If you have some kind of dependency tree that's like: A B C D 1 2 3 i ii Ai i-ii ...and you want to only rebuild Ai and anything that it might rely on, all you do is `dbt build +Ai`. If you want to rebuild B and update anything that relies on it, all you do is `dbt build B+` > You obviously can have modularity with shell scripts Just as kind of a meta point, I think you're assuming that people are introducing a lot of incidental complexity into this problem, but I don't think you're open to the possibility that there's a lot of essential complexity here. I'm talking about DAG tools and the features that hang off of them, and you're saying "hey there's new thing called shell scripting". That's not useful in this conversation. Maybe going forward, let's assume people know about Bash.
  - riku_iki 3y ago
    In my example ETL is divided into modules with clear scope and dependencies, as I believe should happened in software practices. If you have kitchen sink of 1000 tables chaotically connected to each other, then yes, you need DAG management tool to manage chaos..
    - camgunz 3y ago
      > If you have kitchen sink of 1000 tables chaotically connected to each other, then yes, you need DAG management tool to manage chaos. This describes 99% of data teams, so I think we agree!
      - riku_iki 3y ago
        > This describes 99% of data teams, so I think we agree! there is a chance that dbt incentivies data teams to go that route, otherwise they would think about modularity and ownership more.
        camgunz 3y ago
        I'm a Vim user and I've made this point when I rail against IDEs; it's even on the front page today [0]: "5. The belief that complex systems require armies of designers and programmers is wrong. A system that is not understood in its entirety, or at least to significant degree of detail by a single individual, should probably not be built." I think I agree, but I think there are a lot of things you could say about this. What if you're just starting out and you don't know how to efficiently design systems? What if you work in an industry of people under 40 (under 30?) who by definition can't have the kind of experience that would let you design small systems? What if you work for a company that doesn't want to give you the time to figure it out? What if you have a competitor that's just shipping trash and forcing you out of the market with a "we'll clean up the tech debt later" mentality? How do we make tools that are powerful enough to enable us to do hard things, but wise enough to keep us from doing foolish things? Should we rely on our tools to do this? Is this an argument against centralization? Isn't that deeply inefficient? Honestly I struggle with this all the time. I've got the outlines of some ideas and there are some tech communities grappling with the same issues, but I think we're just at the very beginning of a rethink about how we do software. [0]: https://liam-on-linux.dreamwidth.org/88032.html https://liam-on-linux.dreamwidth.org/88032.html

riku_iki 3y ago

You obviously can have modularity with shell scripts: run_all_etl.sh build_internal_dashboard_etl.sh (called from run_all_etl.sh) build_external_dashboard_etl.sh (called from run_all_etl.sh)

itsoktocry 3y ago
How many models are in "run all etl", and what if one fails? You can't see how issues can arise here, with what you describe? I mean, go ahead and write your SQL pipeline in bash. Many of us are using dbt because we see a lot of benefit, we aren't all morons.
- riku_iki 3y ago
  > How many models are in "run all etl", and what if one fails? hundred tables. one fail -> whole pipeline fails to ensure e2e and overall consistency.
camgunz 3y ago
No you misunderstand. If you have some kind of dependency tree that's like: A B C D 1 2 3 i ii Ai i-ii ...and you want to only rebuild Ai and anything that it might rely on, all you do is `dbt build +Ai`. If you want to rebuild B and update anything that relies on it, all you do is `dbt build B+` > You obviously can have modularity with shell scripts Just as kind of a meta point, I think you're assuming that people are introducing a lot of incidental complexity into this problem, but I don't think you're open to the possibility that there's a lot of essential complexity here. I'm talking about DAG tools and the features that hang off of them, and you're saying "hey there's new thing called shell scripting". That's not useful in this conversation. Maybe going forward, let's assume people know about Bash.
- riku_iki 3y ago
  In my example ETL is divided into modules with clear scope and dependencies, as I believe should happened in software practices. If you have kitchen sink of 1000 tables chaotically connected to each other, then yes, you need DAG management tool to manage chaos..
  - camgunz 3y ago
    > If you have kitchen sink of 1000 tables chaotically connected to each other, then yes, you need DAG management tool to manage chaos. This describes 99% of data teams, so I think we agree!
    - riku_iki 3y ago
      > This describes 99% of data teams, so I think we agree! there is a chance that dbt incentivies data teams to go that route, otherwise they would think about modularity and ownership more.
      - camgunz 3y ago
        I'm a Vim user and I've made this point when I rail against IDEs; it's even on the front page today [0]: "5. The belief that complex systems require armies of designers and programmers is wrong. A system that is not understood in its entirety, or at least to significant degree of detail by a single individual, should probably not be built." I think I agree, but I think there are a lot of things you could say about this. What if you're just starting out and you don't know how to efficiently design systems? What if you work in an industry of people under 40 (under 30?) who by definition can't have the kind of experience that would let you design small systems? What if you work for a company that doesn't want to give you the time to figure it out? What if you have a competitor that's just shipping trash and forcing you out of the market with a "we'll clean up the tech debt later" mentality? How do we make tools that are powerful enough to enable us to do hard things, but wise enough to keep us from doing foolish things? Should we rely on our tools to do this? Is this an argument against centralization? Isn't that deeply inefficient? Honestly I struggle with this all the time. I've got the outlines of some ideas and there are some tech communities grappling with the same issues, but I think we're just at the very beginning of a rethink about how we do software. [0]: https://liam-on-linux.dreamwidth.org/88032.html https://liam-on-linux.dreamwidth.org/88032.html