7 ms·
My impression is that "data lake" doesn't refer to the implementation or even the interface. Rather, it refers to an abstract data store where you consolidate a
by lobster_johnson 9y ago
My impression is that "data lake" doesn't refer to the implementation or even the interface. Rather, it refers to an abstract data store where you consolidate all sorts of data: The original source data, as well the intermediate and final outputs of various processing steps. Unlike a database, data lakes just have big blobs of data with no enforced schema.
A distributed file system is just a convenient way of representing it. Let's say you're processing some incoming feed of data. You place new batches of data in this distributed file system. Your processing pipelines consume the data, store some intermediate output, which is then consumed by the next step, etc. — your pipelines will typically transform (clean up, extract, split, etc.) data, potentionally running asynchronously correlations with other data sources (e.g. your feed has street addresses but you want lat/lon coordinates), aggregating it, etc. The steps may store their outputs, and the end of the pipeline will also emit data back into the data lake, from which it can be fed into other, structured data management systems.
One benefit of modeling this as a classical (albeit distributed) file system is that it's familiar to users and can potentially be mounted directly (via something like FUSE) and used interactively, e.g. end-users can manually upload job data or take output data into programs such as R or Mathlab.
Amazon S3 can be considered a data lake, for example.