4 ms·
1. Dealing with biased data 2. Cleaning/understanding data - nearly all data sets I've used have duplicates, missing data, highly anomalous distributions in so
by magneticnorth 8y ago
1. Dealing with biased data
2. Cleaning/understanding data - nearly all data sets I've used have duplicates, missing data, highly anomalous distributions in some fields that indicate we aren't measuring what we think we are, etc. So a lot of my time is spent figuring out what's going on in the data, cleaning up the issues, figure out what subset of the data is reliable, and then dealing with the biases introduced by what is missing or wrong.
3. Dealing with people who don't understand or respect statistics and data science. For example, I've been brought in to "do the analysis" on an "A/B test" where a team didn't appropriately randomize their samples, and also hadn't done a statistical power test beforehand so had an underpowered test anyway, so there was just no hope of validating that their change was an improvement.
- chewxy 8y agoI tried scaling the task of #2 across multiple people recently. It took way longer than had I personally done it myself. I want to know if there is a way to spread the load for this.