6 ms·

Any specific resources you'd recommend on data cleaning, verification, etcetera? I've just started reading this: https://www.amazon.com/Accuracy-Economic-Observ

Any specific resources you'd recommend on data cleaning, verification, etcetera? I've just started reading this: https://www.amazon.com/Accuracy-Economic-Observations-Oskar-Morgenstern/dp/0691003513 https://www.amazon.com/Accuracy-Economic-Observations-Oskar-... . I've seen a few other books on the subject which I'm planning to get into, but I'd be interested if anyone has specific recommendations.

jackschultz 10y ago
Honestly, as another commenter pointed out, seems like an emerging field. Best practices and processes are just being figured out, and I haven't seen any great resources online talking about what to do, especially since most of what you need to do depends on the data set and how you're storing the data, and that can vary widely. Like I recently dealt with finding duplicate song lyrics in my 5000 set of lyrics, and to do that, I just had to google around for StackOverflow answers or random blog posts before I found something that I could adopt and chance for what I had.
chubot 10y ago
Do you use R and the "hadleyverse"? (Or "tidyverse" I think as he prefers?) I'm a programmer by trade but I use R because the people who actually work with data use it, and they write good tools for it... I think there is some confusion in the programming world about this. Programmers work with data, but they don't do it nearly as much as "professionals". Tidy data is a good intro if you're not familiar with it: http://vita.had.co.nz/papers/tidy-data.html http://vita.had.co.nz/papers/tidy-data.html And I would recommend going through other publications by Wickam, all on his site -- they are quite readable.
- elevensies 10y ago
  No, I don't use R, thanks for the reference. Most of what I've done lately has been as part of software development process, so along the lines of validating the effectiveness different techniques for solving a known problem with a smallish test dataset -- typical engineering style optimization. I'm looking to impose more structure on the process.
martingoodson 10y ago
Well this is certainly a good start: https://github.com/Quartz/bad-data-guide https://github.com/Quartz/bad-data-guide
nl 10y ago
Apache NiFi is pretty decent. Eg http://www.slideshare.net/BryanBende/building-data-pipelines-for-solr-with-apache-nifi http://www.slideshare.net/BryanBende/building-data-pipelines...

jackschultz 10y ago

Honestly, as another commenter pointed out, seems like an emerging field. Best practices and processes are just being figured out, and I haven't seen any great resources online talking about what to do, especially since most of what you need to do depends on the data set and how you're storing the data, and that can vary widely. Like I recently dealt with finding duplicate song lyrics in my 5000 set of lyrics, and to do that, I just had to google around for StackOverflow answers or random blog posts before I found something that I could adopt and chance for what I had.

chubot 10y ago

Do you use R and the "hadleyverse"? (Or "tidyverse" I think as he prefers?) I'm a programmer by trade but I use R because the people who actually work with data use it, and they write good tools for it... I think there is some confusion in the programming world about this. Programmers work with data, but they don't do it nearly as much as "professionals". Tidy data is a good intro if you're not familiar with it: http://vita.had.co.nz/papers/tidy-data.html http://vita.had.co.nz/papers/tidy-data.html And I would recommend going through other publications by Wickam, all on his site -- they are quite readable.

elevensies 10y ago
No, I don't use R, thanks for the reference. Most of what I've done lately has been as part of software development process, so along the lines of validating the effectiveness different techniques for solving a known problem with a smallish test dataset -- typical engineering style optimization. I'm looking to impose more structure on the process.

martingoodson 10y ago

Well this is certainly a good start: https://github.com/Quartz/bad-data-guide https://github.com/Quartz/bad-data-guide

nl 10y ago

Apache NiFi is pretty decent. Eg http://www.slideshare.net/BryanBende/building-data-pipelines-for-solr-with-apache-nifi http://www.slideshare.net/BryanBende/building-data-pipelines...