4 ms·
We don't currently have use cases that require heavy transformations (see this blog post I wrote to explain why: https://blog.stitchdata.com/why-our-etl-tool-do
by cm 10y ago
We don't currently have use cases that require heavy transformations (see this blog post I wrote to explain why: https://blog.stitchdata.com/why-our-etl-tool-doesnt-do-transformations-1b2e88c581c3 https://blog.stitchdata.com/why-our-etl-tool-doesnt-do-trans...).
However, since Singer is built around piping data between applications, your suggestion - to code something that sits between taps and targets - makes perfect sense. The whole "flow" would look like:
$ tap-mydatasource | do-aggregations | target-mytarget
We'd be eager to hear from anyone who tries this approach!
- jakestein 10y agoThe only thing I'd add from Chris's blog post is that in the workflow we tend to see is that most of the transformations tend to be done after loading into the destination. For example, in Redshift the transformations could be defined in SQL or Python UDFs.