3 ms·

"I think data scientist is a sexed-up term for a statistician." This statement, given by Silver to the annual meeting of the Joint Statistics Meetings (the mai

by mturmon 13y ago

"I think data scientist is a sexed-up term for a statistician." This statement, given by Silver to the annual meeting of the Joint Statistics Meetings (the main cross-organization stats conference), was guaranteed to be a crowd-pleaser for that audience. Unfortunately for them, it's not really true. The problem is that much of conventional academic statistics consists of proving theorems about model classes. This requires a lot of sophisticated analysis, but has turned rather vacuous. And much conventional applied statistics consists of computing diagnostics based on dubious modeling assumptions. Under pressure in the last 20 or so years from computer science, machine learning, computer vision, Moore's law, and the data avalanche, the discipline has changed, but not fast enough. As a result, a lot of what should be taught and researched in statistics departments has been co-opted by these other disciplines. And many people with a real problem would rather work with a "machine learning" person than a "statistics" person. The best summary of this state of affairs is Leo Breiman's essay (http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?handle=euclid.ss/1009213726&view=body&content-type=pdf_1 http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?ha...). The abstract of this essay is brutal: "There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large, complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools." Breiman was mathematically sophisticated, so it's not that he wasn't able to follow the theory he critiques, it's that he wasn't snowed by detail and could see its lack of relevance to real problems.

bane 13y ago
But I think data scientist, the way it's used today in the job market, is definitely a term for a person who works with real data using software tools and computers through a statistical lens (or a statistician who's comfortable munging data for their analysis). There's simply no term for someone who works with lots and lots of data (not necessarily "big data", I don't like that work) through a more general kind of lens that may or may not involve a statistical approach. When the term "data scientist" first really started to float around I got excited thinking it was describing the latter, when it just ended up describing the former. i.e. A data scientist role is more likely to answer a question like "What percent of people in the White House visitor logs show up in the past 24 months of Reuters News reporting?" vs. "Get me a list of people on the White House visitor logs who've shown up in the past 24 months of Reuters News reporting and build a relationship graph from both the news reports and the visitor logs and merge them." Both require the data munging part of grabbing the visitor log, grabbing the news reports and finding the names in the reports (controlling for various data quality issues). But the "data scientist" approach is to then jump from that onto a statistical answer, while the second approach is to build a social network that might be used for other purposes. These are very different kinds of approaches, with different kinds of outcomes. It's frustrating that a person in the second kind of approach doesn't have a name, "data scientist" would have been perfect, but the field seems to have gone a different direction with the term.
tel 13y ago
The two cultures argument is strong, but so is David Hand's "Illusion of Progress" paper [1] which fits into the debate by highlighting that classifier technologies often touted by machine learning aficionados does not scale to better performance when analyzed carefully. In my experience, improvements in performance tend to come from improvements in domain knowledge. This usually plays out in feature selection, but may involve radical changes in the classifier/model design. The secondary ones are more fun and can more easily be the basis of a PhD. The initial ones are usually swept under the rug when they're brutal and boring, though fundamental changes also form great PhDs. I think it's fairly clear that the success of NLP almost entirely came down to a feature engineering change. I don't know what the ideal Data Scientist/Statistician position is. I think it varies based on many of the implicit needs that only statistics savvy businesses can pick between clearly. The role may require strong programming skills, the ability to build and iterate massive multilevel models of social data, a familiarity on-line classifiers, the ability to graph things nicely in R and D3, the ability to from scratch write a non-linear optimizer over a completely novel space, a familiarity with NLTK, or, finally and perhaps most commonly, the ability to pander bullshit about the benefits of machine learning. I love the field and that's why I tend to side with the statisticians who want to really figure out what's needed to do an objectively good job drawing inference from data. Many in the new generation of statisticians are quite successfully crossing the cultural divide. A smaller cohort are brushing up at least their R coding skills to a suitable degree. Some already have an outside competency in programming. I'd like these people to eventually form the core of what real "data science" is. [1] http://arxiv.org/pdf/math/0606441.pdf http://arxiv.org/pdf/math/0606441.pdf
- mturmon 13y ago
  That paper by David Hand is good, I had forgotten it. (Even though I presented it to a reading group at my work...). He points out a lot of ways the standard "supervised learning, iid feature vectors, moderate dimensionality, moderate n" problem setting is a theoretical construct that does not encompass enough of the real-world problem setting. And he highlights the fact that an epsilon improvement in error rate is not that remarkable. So, I agree that his paper was smart as a critique of a certain culture within ML practice (and publications) of totally eliminating all problem context, and just saying "Give me a bag of labeled feature vectors -- I will build a classifier with an error rate better than your classifier, and publish the result." But that is largely a straw man -- critiquing ML hackery is worthwhile and salutary, but it's going after small game. The random forest, support vector machine, and deep learning approaches should have been developed in Statistics. Why weren't they?

by mturmon 13y ago

bane 13y ago

But I think data scientist, the way it's used today in the job market, is definitely a term for a person who works with real data using software tools and computers through a statistical lens (or a statistician who's comfortable munging data for their analysis). There's simply no term for someone who works with lots and lots of data (not necessarily "big data", I don't like that work) through a more general kind of lens that may or may not involve a statistical approach. When the term "data scientist" first really started to float around I got excited thinking it was describing the latter, when it just ended up describing the former. i.e. A data scientist role is more likely to answer a question like "What percent of people in the White House visitor logs show up in the past 24 months of Reuters News reporting?" vs. "Get me a list of people on the White House visitor logs who've shown up in the past 24 months of Reuters News reporting and build a relationship graph from both the news reports and the visitor logs and merge them." Both require the data munging part of grabbing the visitor log, grabbing the news reports and finding the names in the reports (controlling for various data quality issues). But the "data scientist" approach is to then jump from that onto a statistical answer, while the second approach is to build a social network that might be used for other purposes. These are very different kinds of approaches, with different kinds of outcomes. It's frustrating that a person in the second kind of approach doesn't have a name, "data scientist" would have been perfect, but the field seems to have gone a different direction with the term.

tel 13y ago

The two cultures argument is strong, but so is David Hand's "Illusion of Progress" paper [1] which fits into the debate by highlighting that classifier technologies often touted by machine learning aficionados does not scale to better performance when analyzed carefully. In my experience, improvements in performance tend to come from improvements in domain knowledge. This usually plays out in feature selection, but may involve radical changes in the classifier/model design. The secondary ones are more fun and can more easily be the basis of a PhD. The initial ones are usually swept under the rug when they're brutal and boring, though fundamental changes also form great PhDs. I think it's fairly clear that the success of NLP almost entirely came down to a feature engineering change. I don't know what the ideal Data Scientist/Statistician position is. I think it varies based on many of the implicit needs that only statistics savvy businesses can pick between clearly. The role may require strong programming skills, the ability to build and iterate massive multilevel models of social data, a familiarity on-line classifiers, the ability to graph things nicely in R and D3, the ability to from scratch write a non-linear optimizer over a completely novel space, a familiarity with NLTK, or, finally and perhaps most commonly, the ability to pander bullshit about the benefits of machine learning. I love the field and that's why I tend to side with the statisticians who want to really figure out what's needed to do an objectively good job drawing inference from data. Many in the new generation of statisticians are quite successfully crossing the cultural divide. A smaller cohort are brushing up at least their R coding skills to a suitable degree. Some already have an outside competency in programming. I'd like these people to eventually form the core of what real "data science" is. [1] http://arxiv.org/pdf/math/0606441.pdf http://arxiv.org/pdf/math/0606441.pdf

mturmon 13y ago
That paper by David Hand is good, I had forgotten it. (Even though I presented it to a reading group at my work...). He points out a lot of ways the standard "supervised learning, iid feature vectors, moderate dimensionality, moderate n" problem setting is a theoretical construct that does not encompass enough of the real-world problem setting. And he highlights the fact that an epsilon improvement in error rate is not that remarkable. So, I agree that his paper was smart as a critique of a certain culture within ML practice (and publications) of totally eliminating all problem context, and just saying "Give me a bag of labeled feature vectors -- I will build a classifier with an error rate better than your classifier, and publish the result." But that is largely a straw man -- critiquing ML hackery is worthwhile and salutary, but it's going after small game. The random forest, support vector machine, and deep learning approaches should have been developed in Statistics. Why weren't they?