At work we built something that's maybe halfway in between the data lake and data warehouse. It's working well for us. The basic setup: - All data is CSV or

At work we built something that's maybe halfway in between the data lake and data warehouse. It's working well for us. The basic setup: - All data is CSV or json-document-per-row text files on Amazon S3. - We have a Django web application that keeps track of metadata (which dataset lives in which S3 bucket/folder, who uploaded it, the names of the columns in that dataset, and their types). - The REST API in the Django web piece can provide temporary signed S3 URLs that can allow anyone in the company to create a new dataset and upload the files to S3. - The REST API also provides Hive and Redshift "CREATE TABLE" commands for all datasets (which it builds from the columns/types data stored in the DB). We've talked several times about open sourcing it but haven't gotten around to making that happen yet.