4 ms·
How much data are we talking about? If the data is small enough, then an API is better. If you need to read a lot of data GBs/hour, then you probably want to re
by mateo411 7y ago
How much data are we talking about? If the data is small enough, then an API is better. If you need to read a lot of data GBs/hour, then you probably want to read data through the data lake.
However, it's still possible to define schemas, version, and set SLAs on the datasets in the data lake. If you don't do that, then you are going to have to fix this whenever the upstream dataset changes. This is even better if the upstream dataset is produced by a team and they have no idea that they are other people consuming their datasets. Welcome to the Wild West of Data Engineering.
- kyllo 7y agoYeah, my first two questions were: "Why don't you get the data from the upstream team's API? Have you discussed this design with them?" It also happened to me once before--a production system took a dependency on one of my streams in the data lake without my knowledge, and no SLA, then when I made a change to it, they came out of nowhere, blowing me up with e-mails and meeting requests. I told them to go pound sand.