3 ms·

I dont know if the commentator's assertions are correct, but my anecdotal experience follows: A statistician asks me if adding a feature could make a model wors

I dont know if the commentator's assertions are correct, but my anecdotal experience follows: A statistician asks me if adding a feature could make a model worse (R^2), and I am like of course! He gets testy, and snidely chimes back 'i don't know why you think that could happen'. And I think 'I don't know why you don't think you can over fit data...' then it hit me, statisticians thinking revolves around running the model on the entire data set, where my thinking revolves around how it performs on test data

aub3bhat 9y ago
There is a good paper by Leo Breiman [1] which discusses the exact issue that you mentioned. E.g. Statisticians believe that there is a "True" model that generates the data, and the errors observed are merely noise. ML on the other hand does not assumes existence of a "True" model, the assumption is that only the Data is source of the truth and any model that predicts the data with lowest error is preferable subject to performance on test/cross-validation etc. This is a powerful approach and distinguishes ML as separate field from statistics. My favorite example is predicting prices of real estate. A statistician will build a multi-level model that takes into account various effects (Zip code level, city level, school district level, year/month of acquisition etc.) and then build a regression model. The errors then would then be simply noise, an ML approach would be simply using weighted K-Nearest Neighbors, with geographic location as part of distance metric. Sure there are no effects to adjust but the K-NN regression model can account for difficult to capture quirks of geography by being able to represent local non-linear decision surface. http://projecteuclid.org/euclid.ss/1009213726 http://projecteuclid.org/euclid.ss/1009213726

3 ms·

I dont know if the commentator's assertions are correct, but my anecdotal experience follows: A statistician asks me if adding a feature could make a model wors

by autokad 9y ago

aub3bhat 9y ago
There is a good paper by Leo Breiman [1] which discusses the exact issue that you mentioned. E.g. Statisticians believe that there is a "True" model that generates the data, and the errors observed are merely noise. ML on the other hand does not assumes existence of a "True" model, the assumption is that only the Data is source of the truth and any model that predicts the data with lowest error is preferable subject to performance on test/cross-validation etc. This is a powerful approach and distinguishes ML as separate field from statistics. My favorite example is predicting prices of real estate. A statistician will build a multi-level model that takes into account various effects (Zip code level, city level, school district level, year/month of acquisition etc.) and then build a regression model. The errors then would then be simply noise, an ML approach would be simply using weighted K-Nearest Neighbors, with geographic location as part of distance metric. Sure there are no effects to adjust but the K-NN regression model can account for difficult to capture quirks of geography by being able to represent local non-linear decision surface. http://projecteuclid.org/euclid.ss/1009213726 http://projecteuclid.org/euclid.ss/1009213726