4 ms·

Isn't this the whole point of a data lake?

• BrentOzar 7y ago
  > Isn't this the whole point of a data lake? Yes, but data lakes don't fill themselves. Each team has to be responsible for exporting every transaction to the lake, either in real time or delayed, and then the reporting systems have to be able to combine the different sources/formats. If each microservice team expects to be able to change their formats in the data lake willy-nilly, bam, there breaks the report again.
  • iamEAP 7y ago
    This problem exists in monolithic data stores and codebases too. It's not as if independent teams have absolute sovereignty over their table schemas. Schemas evolve as the needs of the product change, and that evolution will always outpace the way the business looks at the data. The best way I've seen to deal with this is to handle this at report query-time (e.g. pick a platform that can effectively handle the necessary transformations at query-time, rather than at load-time).