3 ms·
> using rigorous languages and schemas And what value does it add? A vast majority of companies are working with < 1TB of data that sits neatly in a single cl
by tomnipotent 5y ago
> using rigorous languages and schemas
And what value does it add?
A vast majority of companies are working with < 1TB of data that sits neatly in a single cloud database. Python and tools like dbt are fantastic for a huge class of problems without compromising workflow velocity, and pushing transformations into SQL removes most Python-bound performance constraints.
Changing singer.io to require Thrift or protobuf schemas isn't going to add the value you think it is. How data is shuffled between systems is considerably less important and time consuming than figuring out how to put that data to work.
- glogla 5y agoSinger and dbt are different - dbt orchestrates stuff (evaluates macros and runs SQLs) while Singer actually has the data flow through it. So rewriting Singer in something fast (I guess Java since even the most obscure database has JDBC driver for it) would definitely help. It would only help once you start shuttling around terabytes.
- tomnipotent 5y agoThese tools were not built for multi-tb workloads, and pointing out that as a deficiency when it was clearly not a design goal is a misleading argument.