3 ms·
The interesting point here is that the unbundling of BI is resulting in tools for every occasion. What I'm seeing on my end is that the "modern data stack" is s
by blakeburch 5y ago
The interesting point here is that the unbundling of BI is resulting in tools for every occasion. What I'm seeing on my end is that the "modern data stack" is starting to become very fragmented as a result.
Sure, each of these tools are better suited to accomplishing a very specific task with your data. But the problem is that now these tools aren't talking to each other. No one has a single place to view their data end-to-end. No one can show me every automated process in their organization that relies on a specific table. Everything is tied together with trust. Trust that the Data Engineering team won't deploy bad code. Trust that the API won't change how it returns data. Trust that the data actually loaded completely.
This trust-based system means that one problem during ingestion/cleaning ends up spelling disaster for all of the downstream ML models, reports, data extracts, etc. and causes a day worth of headaches to resolve.
If BI used to hold everything together under one roof, we need something to make these tools talk to each other. The only solution, as I see it, is to build out better data orchestration that effectively glues these tools together and lets you see the "big picture" with your data.
- CRConrad 5y agoDon't worry, you won't have to invent a whole new thing: There is a whole sub-industry of IT called Data Warehousing that does exactly this.