4 ms·
Singer and dbt are different - dbt orchestrates stuff (evaluates macros and runs SQLs) while Singer actually has the data flow through it. So rewriting Singer i
by glogla 5y ago
Singer and dbt are different - dbt orchestrates stuff (evaluates macros and runs SQLs) while Singer actually has the data flow through it. So rewriting Singer in something fast (I guess Java since even the most obscure database has JDBC driver for it) would definitely help.
It would only help once you start shuttling around terabytes.
- tomnipotent 5y agoThese tools were not built for multi-tb workloads, and pointing out that as a deficiency when it was clearly not a design goal is a misleading argument.