5 ms·
Azure Data Lake Store: a hyperscale distributed file service for analytics
- theatraine 9y agoI'm a data scientist at Microsoft and use the precursor to Azure Data Lake (internally called Cosmos) daily. It's a remarkably powerful and well-designed system.
- zitterbewegung 9y agoWhat do you use it for? How and or why is it better than other systems? Are there any trade offs compared to other systems?
- jchanimal 9y agoIs the Cosmos you reference related to https://azure.microsoft.com/en-us/services/cosmos-db/ https://azure.microsoft.com/en-us/services/cosmos-db/
- bjg 9y agoNo... it's really confusing :)
- sargun 9y agoIs this really Cosmos? Does that mean SCOPE queries are coming?
- alyx 9y agoSCOPE is for Cosmos what USQL is for Azure Data Lake Analytics There are a lot of similarities between Scope and USQL, but USQL is also a cleaner (and in some ways more rigid) implementation.
- sbarre 9y agoSerious question: is a "data lake" just a really really really scalable filesystem? I see it described in many different ways but ultimately that's what it sounds like to me. Obviously I'm sure there's massive effort involved under the hood in presenting it as "just a filesystem" to the end-user (or end-process), but am I missing anything else?
- tomnipotent 9y agoA "data lake" just means you've extracted and consolidated data from external systems and left it as-is - warts and all. Upstream data warehouses/marts/cubes etc. would be built from the data lake.
- sbarre 9y agoYeah I understand what it is conceptually, I guess I was more curious about what it actually implements.. Sorry I wasn't clear about that.
- lobster_johnson 9y agoMy impression is that "data lake" doesn't refer to the implementation or even the interface. Rather, it refers to an abstract data store where you consolidate all sorts of data: The original source data, as well the intermediate and final outputs of various processing steps. Unlike a database, data lakes just have big blobs of data with no enforced schema. A distributed file system is just a convenient way of representing it. Let's say you're processing some incoming feed of data. You place new batches of data in this distributed file system. Your processing pipelines consume the data, store some intermediate output, which is then consumed by the next step, etc. — your pipelines will typically transform (clean up, extract, split, etc.) data, potentionally running asynchronously correlations with other data sources (e.g. your feed has street addresses but you want lat/lon coordinates), aggregating it, etc. The steps may store their outputs, and the end of the pipeline will also emit data back into the data lake, from which it can be fed into other, structured data management systems. One benefit of modeling this as a classical (albeit distributed) file system is that it's familiar to users and can potentially be mounted directly (via something like FUSE) and used interactively, e.g. end-users can manually upload job data or take output data into programs such as R or Mathlab. Amazon S3 can be considered a data lake, for example.
- viblo 9y agoCould someone explain the differences between Azure Data Lake and Azure Blob Storage?