3 ms·

Cool article, thanks. I used to spend 90% of my downtime trying to improve my stats and machine learning knowledge, but in the last couple years I've come to re

by rm999 14y ago

Cool article, thanks. I used to spend 90% of my downtime trying to improve my stats and machine learning knowledge, but in the last couple years I've come to realize how much my lack of proper engineering was hurting me. A (flawed) analogy: if data science is storytelling, stats is the story and engineering is the words you use to tell it. You need to do well at both to effectively tell your story. >...good engineering going out the window quickly with elaborate ensembling. This is one of my criticisms of data mining contests (sorry kaggle!). When I was in grad school I liked doing these contests to get practical experience - my last company actually recruited me through one. But as they got more popular I found the best engineers and data scientists stopped having as much of a chance of winning. Good modelers get 90% of the way there and then are beaten by impractical solutions that would get someone fired from a job.

numlocked 14y ago
It's an interesting criticism and there are competition properties that we've identified that can lead to overly complex models. An obvious one, for instance, is running a competition for a very long period of time. A more subtle factor is the degree to which the context of the data has been stripped away - with very little context (just an anonymized feature matrix) it's more likely that victory will go to someone who can squeeze every bit of performance out of an ensemble of ML techniques. But with good context, it's more likely that feature engineering and insight will lead to victory. Many times it is in fact the well-organized, well-engineered solutions that win. And they are often elegant and intuitive because the data scientist could iterate quickly and try out lots of ideas. Though the crazy ensemble does sometimes eek out a victory, it's not an inherent property of competitions by any means. One final note: I'm not sure if this is widely known, but impractical solutions are actually just fine for a certain set of competitions. Some competitions are designed to identify the "data frontier" - the limit of what's possible in a dataset - rather than getting a model into production.

by rm999 14y ago

numlocked 14y ago

It's an interesting criticism and there are competition properties that we've identified that can lead to overly complex models. An obvious one, for instance, is running a competition for a very long period of time. A more subtle factor is the degree to which the context of the data has been stripped away - with very little context (just an anonymized feature matrix) it's more likely that victory will go to someone who can squeeze every bit of performance out of an ensemble of ML techniques. But with good context, it's more likely that feature engineering and insight will lead to victory. Many times it is in fact the well-organized, well-engineered solutions that win. And they are often elegant and intuitive because the data scientist could iterate quickly and try out lots of ideas. Though the crazy ensemble does sometimes eek out a victory, it's not an inherent property of competitions by any means. One final note: I'm not sure if this is widely known, but impractical solutions are actually just fine for a certain set of competitions. Some competitions are designed to identify the "data frontier" - the limit of what's possible in a dataset - rather than getting a model into production.