4 ms·
That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later w
by jlmorton 7y ago
That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice.
Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing with these schema changes for 5% of the data is easier than carefully planning a public schema for 100% of it.
It can be helpful to include some small amount of metadata in the export though, with things like the source system name, date & time, # of records, and a schema version. Schema version could easily be the latest migration revision, or something like that.
- te_chris 7y agoThe data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out
- lolc 7y agoBut if I haven't spent the effort to extract it, do I really own it? Let me argue that I don't have it because all my implemented queries turn up none of your data. You wouldn't tax me on gold that hasn't yet been extracted, would you? (End of joke.)
- ska 7y agoBut if I haven't spent the effort to extract it, do I really own it? If you collected it, you are responsible for it.
- yesbabyyes 7y agoWhat I think you'd typically do is put different data under different keys/paths, so that red is personally identifiable data, yellow contains pointers to such data, and green is just regular data. You could have a structure like s3://my-data-lake/{red|yellow|green}/{raw|intermediate}/year={year}/month={month}/day={day}/source={system}/dataset={table} Then you just don't keep red data for longer than 30 days.