3 ms·
I have seen it implemented in one of the financial services companies. Both data warehouse and and data lake type databases lived on MS SQL servers (with the us
by mynegation 11y ago
I have seen it implemented in one of the financial services companies. Both data warehouse and and data lake type databases lived on MS SQL servers (with the usual production/testing division), and nightly batch processes performed clean up from data lake to data warehouse. It kind of worked.
The usual problems with these are:
(1) getting hold of external clean up functionality. Sometimes it was stored with the database itself as stored procedures, but sometimes it was not. Remedy: make sure clean up functionality is readily available.
(2) dealing with new kinds of "dirt" (i.e. problematic data in the data lake). Remedy: this is unavoidable. The best you can do is to insert lots and lots safety checks and diagnostics into your clean up programs