4 ms·
I'm a data engineer at a large corporation. At my current company, we use Pentaho to extract and transform our data from Oracle daily. The transformed data is l
by vinhdp 6y ago
I'm a data engineer at a large corporation. At my current company, we use Pentaho to extract and transform our data from Oracle daily. The transformed data is loaded to a staging database where we then model it to a star-schema. Then the final results are bulk loaded to an on-prem DW. The process takes hours to complete.
I’m interested in moving to a model that continually extracts changed data to a data lake, then using the power of a cloud data warehouse to read those files and perform the transformations and modeling in SQL. I guess that's the ELT concept that you mentioned in the book's summary.
Goal being to reduce the latency and allow for the possibility of more frequent batches, as well as making the process more accessible to my team with strong SQL skills and being able to adapt faster to changing business needs.
This book looks like a good foray for me to get a glimpse into those new process. Thanks for putting it together.
- huy 6y agoI'm one of the authors of the book. Yes you're right. The book outline the transition from the "old world of BI" to the "new world of BI". If you read through the book, you'll see the biases clearly stated there: - ELT over ETL - Data Modeling is crucial as part of BI workflow - Cloud DW over on-premise DW (but this depends on your org's requirements) - SQL reporting (Redash, Metabase, Looker, Holistics) over non-SQL reporting (Tableau, Qlik)
- mritchie712 6y agoI'd toss SeekWell[1] in the "new world" category, but we've taken a different approach. Instead of forcing people to use a whole separate platform for BI, we decided to tightly integrate with the tools people were already going to for data (e.g. Google Sheets, Excel, Slack, etc.). We've found teams stay better informed when the data is in places they're already hanging out. [1] https://seekwell.io/ https://seekwell.io/