4 ms·
DWs like Snowflake certainly share some features you'll find in traditional RDBMSs, but I'm not sure I would classify it as one. https://www.quora.com/Is-Snowfl
by chrisjc 6y ago
DWs like Snowflake certainly share some features you'll find in traditional RDBMSs, but I'm not sure I would classify it as one. https://www.quora.com/Is-Snowflake-a-RDBMS https://www.quora.com/Is-Snowflake-a-RDBMS
> That's what SQL is good for, why (if we're already in an RDBMS) do I need another tool on top to do this?
Writing, versioning, testing, deploying, monitoring all that complicated SQL is very challenging. This is what DBT is good for. DBT doesn't replace SQL, it simply augments it by allowing you templatize, modularize, figure out all the inter-dependencies, etc...
> Of course, that's what a DW does as it pulls in new data. But that is what SQL also does rather well, so what does this tool bring to the party?
Sounds like your talking more about ETL, but from what I understand DBT is really useful for what happens after ingestion. Transforming, refining, aggregating, etc from one internal DW source to another. This is where you would want to use SQL, but again, DBT doesn't replace SQL, it augments it.
----
BTW, I've never used DBT and have only really looked into it in the last 24hrs since we're really struggling with the pain of managing all of our ETL (table->table) SQL. I really see where it could make our lives a lot simpler, but it doesn't seem to provide the functionality that we need for our workloads.
We're using Snowflake change streams to build refined, summary and aggregation tables off an ingestion table (Kafka Connect to SF). We use change streams on that ingestion table to track new data that we can refined, summarized and aggregated into final tables in a near real-time fashion that's used for analytics and reporting. Change streams (and tasks and procedures) is not something that DBT seems to be ready for, but it's probably understandable since they're trying to provide functionality to more DWs than just SF.
- throwaway_pdp09 6y agoSnowflake is per your link an RDBMS. Where it lives and on what storage subtrate it operates from is irrelevant. On your recommendation I'll have a look at DBT, thanks. Whatever you're doing at work sounds really complex, possibly worsened by sheer quantity of data. I wish you luck with it!
- throwaway_pdp09 6y agoI've looked at it and it seems to be a gross mashup of text processor and SQL. And I really don't see the value of it at all.