8 ms·
> Extracting the data should be the most challenging aspect of an ETL pipeline. Why should this be difficult? It’s the easiest part. You run SELECT * and you
by mulmen 2y ago
> Extracting the data should be the most challenging aspect of an ETL pipeline.
Why should this be difficult? It’s the easiest part. You run SELECT * and you’re done.
The difficult part is transforming all the disparate upstream systems and their evolving schemas into a useful analytical model for decision support.
- bob1029 2y agoNot all data lives in a SQL database. Much of the extraction code I write does things like loading flat files from unusual sources and querying APIs. If the source data is already in a SQL store, then the solution should be obvious. You don't need any other tools to produce the desired view of the business at that point. Transforming for an upstream schema is a select statement per target table. This doesn't need to be complicated.
- mulmen 2y agoYeah I extract a lot of data out of Dynamo. It’s still the easiest part. Change capture just isn’t complicated. You need some basic constructs and then you’re golden. The data mart design phase is orders of magnitude more effort.