3 ms·

Sort of. The issue is coupling data with sql databases doesn’t play well with even basic inferential statisics so you end up replicating data based on analytics

by kfk 5y ago

Sort of. The issue is coupling data with sql databases doesn’t play well with even basic inferential statisics so you end up replicating data based on analytics use cases. Having parquet files in a cheap storage system that can be queried with sql, python, R, etc. is very convenient. What I don’t get is why so little investments are going in making data lakes safer and easier to govern with proper access controls

manigandham 5y ago
You can run those queries straight from relational databases too, since everything ends up in the same dataframe structure for processing in the end. There are tons of data source adaptors now and all the modern OLAP databases support various deeper integrations to run stats/ML directly.