4 ms·

Thanks for sharing!! Author here: yeah I ended up insisting a bit much on dbt as a basis although the method I'm describing in the article can definitely be ap

by b-luu 4y ago

Thanks for sharing!! Author here: yeah I ended up insisting a bit much on dbt as a basis although the method I'm describing in the article can definitely be applied to any kind of modeling framework (or lack of) Anyway, please let me know what you think of the general method and the bottom-up VS top-down approach ;-)

sails 4y ago
Good detailed breakdown of how to use dbt, but I'm not sure about the data modelling part (the specific approach suggested), as it seems possibly contradictory: > In contrast, the approach I want to outline here is primarily bottom-up, in order to: - deliver value and insights as quickly as possible - not pile up too much (data modeling) technical debt along the way. > The goal will be to aim towards either: - a dimensional model/star schema - or a limited collection of "One Big Tables" (or any other top-down theory you’re most comfortable with...) I get (and agree) that you want to start off organic and get things moving with some integrations, but I quite strongly feel that with the suggested approach (which feels to me like - just start with the dbt part, don't worry about the final design part), especially if you don't decide _up front_ what capabilities you, your team, your client have, and choose an approach based on that, you'll likely end up with something of a technical debt mess. My suggestion would be, unless otherwise advised, to just adopt the dimensional model, put that in the hands of your users early, and start getting feedback. In the meantime you can do exactly what is described, but with a feedback loop kicked off much earlier than you would have otherwise. edit - maybe contradictory is not correct, the suggestion stands though :)
- b-luu 4y ago
  Thanks for your suggestion. I have nothing against dimensional modeling per-say, just the uptime effort (and initial feedback lag) that it generally brings with it in the beginning. Another issue I've observed with teams trying to formalize their modeling "too" soon is a confusion in the models created. I find that it's sometimes difficult for teams to understand what should be in each model, how to separate the different dimensions, etc. Hence my emphasis on starting naively in the beginning and then aiming for that dimensional model when things start becoming clearer...
  - sails 4y ago
    Agree it is very sensitive to the context. The outcomes are a mixed bag, but my biggest experienced failing was on a project starting naively with OBT and waiting for things to become clearer, it did indeed soon became clear that we should have taken a more methodical approach, as we had the dual problem of huge dependencies and a very brittle data model! (I don't think there is an easy answer here btw, I am hoping for someone to describe a detailed and relatively generalisable playbook, hopefully soon!)

by b-luu 4y ago

sails 4y ago

Good detailed breakdown of how to use dbt, but I'm not sure about the data modelling part (the specific approach suggested), as it seems possibly contradictory: > In contrast, the approach I want to outline here is primarily bottom-up, in order to: - deliver value and insights as quickly as possible - not pile up too much (data modeling) technical debt along the way. > The goal will be to aim towards either: - a dimensional model/star schema - or a limited collection of "One Big Tables" (or any other top-down theory you’re most comfortable with...) I get (and agree) that you want to start off organic and get things moving with some integrations, but I quite strongly feel that with the suggested approach (which feels to me like - just start with the dbt part, don't worry about the final design part), especially if you don't decide _up front_ what capabilities you, your team, your client have, and choose an approach based on that, you'll likely end up with something of a technical debt mess. My suggestion would be, unless otherwise advised, to just adopt the dimensional model, put that in the hands of your users early, and start getting feedback. In the meantime you can do exactly what is described, but with a feedback loop kicked off much earlier than you would have otherwise. edit - maybe contradictory is not correct, the suggestion stands though :)

b-luu 4y ago
Thanks for your suggestion. I have nothing against dimensional modeling per-say, just the uptime effort (and initial feedback lag) that it generally brings with it in the beginning. Another issue I've observed with teams trying to formalize their modeling "too" soon is a confusion in the models created. I find that it's sometimes difficult for teams to understand what should be in each model, how to separate the different dimensions, etc. Hence my emphasis on starting naively in the beginning and then aiming for that dimensional model when things start becoming clearer...
- sails 4y ago
  Agree it is very sensitive to the context. The outcomes are a mixed bag, but my biggest experienced failing was on a project starting naively with OBT and waiting for things to become clearer, it did indeed soon became clear that we should have taken a more methodical approach, as we had the dual problem of huge dependencies and a very brittle data model! (I don't think there is an easy answer here btw, I am hoping for someone to describe a detailed and relatively generalisable playbook, hopefully soon!)