Proper featurization is a hugh part of working with real data. Datasets in contests tend to have already been analyzed enough to remove all features that are co

Proper featurization is a hugh part of working with real data. Datasets in contests tend to have already been analyzed enough to remove all features that are correlated, caused by or predictive of but irrelevant to the target feature. For example, early analysis of a "fresh" data set involves a lot more "'of course length of hospital stay' correlates with 'had disease'" or "'Phone number' is a unique identifier in this data so of course it shows up most often in random forest predictor" moments. Putting together weighted ensembles in R of a clean data set is basically manual labor but doing the same analysis at scale involves a lot of of nuances that should be considered data science. In particular you need to be able to determine if a problem can be split into mostly independent parallelizable sub problems (which also often requires domain knowledge) or reduced/relaxed into something that has a well established optimized solution (like matrix decomposition). And finally you need to be able to determine if your prototype and the production code are converging to the same results and debug it if it isn't which is non trivial with stochastic algorithms.