3 ms·
It's basically a reliable, horizontally scalable object store + a collection of data storage and processing engines. So amongst the cloud providers, AWS calls
by shrikant 8y ago
It's basically a reliable, horizontally scalable object store + a collection of data storage and processing engines.
So amongst the cloud providers, AWS calls a combination of S3 + Glue + Athena (for example) a "data lake", where S3 is the object storage which can store data in various formats, and Glue and Athena are used to transform/process/query the data. See a more detailed article/guide here: https://aws.amazon.com/lake-formation/ https://aws.amazon.com/lake-formation/
If you didn't want to put anything into the cloud and keep all your services on-premise, a local Hadoop cluster could be a data lake, for example using HDFS + Zookeeper + YARN + Hive.
[This is a huge over-smiplification because it's late and I really should be going home :)]
- mucio 8y agoTo storage and compute I would also add metadata, this is why most data lake projects end up in data swamps. When people just dump data in their storage they end having really hard time sharing them in their organization.