This is great timing because I'm working on a similar system for a research center. The data lake is a great idea for researchy or data-sciencey shops where th

This is great timing because I'm working on a similar system for a research center. The data lake is a great idea for researchy or data-sciencey shops where the analysis is the bread and butter. What happens here is that people get data, clean it themselves in ways only they care about, and then go about their business. The problem is there is no one centralized repository of good, clean datasets. Instead it's spread across people, who (unwittingly) hoard domain knowledge on them, and then are now stuck with doing anything else that's required with it, as well as passing it off to other people. Then these people leave. So people keep repeating work in a non-DRY way. Or they do clean it for everyone else to use, but the format and naming isn't standard across datasets, or is spread across weird locations, which again complicates the workflow. Instead, what we want is an internal tool to be able to search and browse data (think data.gov or CKAN), to have a native command to do that from within their tool of choice (R, STATA, matlab), and to have a process that goes with this. The hope for the end result is a streamlined process where when people wonder if something is viable or interesting, they can find a load a dataset within seconds and test a theory, as opposed to having to ignore the impulse because it's too much of a PITA.