7 ms·

Nate Silver to would-be data scientists: Don’t sweat the degree, do the work

39 points by cdl 13y ago

jacques_chester 13y ago
Original source: http://blogs.hbr.org/2013/09/nate-silver-on-finding-a-mentor-teaching-yourself-statistics-and-not-settling-in-your-career/ http://blogs.hbr.org/2013/09/nate-silver-on-finding-a-mentor...
- cruise02 13y ago
  Is it just me, or is the gigaom article basically blogspam?
  - jacques_chester 13y ago
    Saying that something is blogspam can be surprisingly controversial hereabouts. I find it safer to passive aggressively link to the original source.
    - cruise02 13y ago
      Sage advice. It's more productive to find the link and post it than it is to just complain about it, so I guess I should follow your lead. :)
clubhi 13y ago
This is the same with just about everything. When I started learning Clojure I forced myself to stop reading.
carlob 13y ago
There is something that somewhat irks me about the whole domain of data science. I still can't quite put my finger on it, but I think it's a mixture of lack of rigor, handwaving arguments. Maybe if it were called data journalism, or data storytelling rather than science I would somewhat lower my expectations and enjoy it more fully, but in my experience doing science and doing data science are not really close.
- jonahx 13y ago
  I think you could make the same argument about computer science, especially relating it to the actual practice of professional programming, which resembles pop culture more than science. And while both fields have mathematical underpinnings, the foundations of data science -- probability and statistics -- have deeper roots and a longer history. I guess I'm saying you shouldn't impugn the field itself because of its popular practice.
  - carlob 13y ago
    I don't think what most programmers do is computer science, more like software engineering.
    - dllthomas 13y ago
      "more like", yes...
- ig1 13y ago
  Data science is about building testable models, which fundamentally is what science is about. What probably makes you uncomfortable is that Data Science (unlike most scientific fields) doesn't have a requirement for the models to make intuitive sense to humans.
pvnick 13y ago
I agree whole heartedly. When I first started playing around with data, despite my lite statistics background, I had no idea what I was doing and no amount of education would have prepared me for noisey, real life data sets. I had to become proficient with SQL and a certain type of problem solving and hacker mentality before I was able to do anything useful. What's interesting now though is that since returning to college and taking more advanced courses in probability and statistics, I now have a "practical" background that allow me to see real life use cases for almost everything I'm learning (oh wonderful Bayes theorem!) that is a huge advantage.
kartikkumar 13y ago
I find the don't-go-to-school-just-do-it mentality that pervades tech blogs, article, discussions a bit short-sighted. I fully appreciate the importance of getting "hands-on experience" in addition to book learning, but what I don't get is the black-and-white perspective that formal education gives you NO EXPERIENCE WHATSOEVER. As a PhD candidate, my perspective is quite definitely skewed, but my experience in school so far has not given me the impression that all I've been doing is book learning with no transferrable, real-world skills to work with data. I've had to work with plenty of data sets during my Masters and PhD research. I firmly believe that the experience I've gained from applying what I've learned in the classroom and from surveying existing literature on data mining and statistical methods stands me in good stead to tackle "real data" (whatever that means). Maybe I've got the wrong end of the stick, but I think that something more than the "odd stats class" can be of value to a budding data scientist ...
- ThomPete 13y ago
  Don't you find it odd that education start with the abstract to have you end up in reality. Data science is not about well structured data, it's not about learning what others do. I It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data. First experiment, learn to play around, explore, then apply theory.
  - kartikkumar 13y ago
    Actually, I don't think the process of going from abstract to reality is that odd. The reason I say that is because I think understanding the abstract prepares you to be flexible. In terms of data science, the way I see this is that by experimenting without having a grounding in the algorithms, technologies etc. means that you can only really ever get good at the specific implementations you're working with. The abstract understanding is precisely what allows you to be versatile in adopting new technologies and seeing the implementation for what it is: one concrete version of the underlying abstract ideas. There's nothing that says that formal education only prepares you to work with well-structured data. In fact, I think that in grad school you are challenged to work at the cutting edge almost all the time. Granted, a lot of classroom examples at the undergrad level are perhaps trivial, but my experience is that grad school doesn't by definition only serve to prep you with pointless examples. > It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data. I totally agree, and I believe that school is not at odds with this. I'm working in the fields of planetary science and space engineering at the moment, and I am constantly stimulated to experiment with data, be creative and understand correlations. Maybe for some reason my experience is just not a good 3-sigma example ...
hooande 13y ago
It's worth pointing out that Nate Silver is a rare individual. He's been wildly successful with data science despite not having a deep academic background, which is not the common case. His story is a useful data point though. Do you need multiple degrees to master data science? Hell no. The practice of statistical inference requires only a few key skills. The important part is understanding how to think in terms of probability and algorithms, which doesn't require a rigorous understanding of proofs and formulae. An advanced academic degree is useful but not required. The world is looking for people who can consistently get the right answer, regardless of their background. The other side to this is that a degree will be required to work for wall st, google, harvard, etc. Degrees still matter and social signalling and so on and so forth. We all can't be as well known and popular as Nate Silver and at some point institutions need a way of establishing that a particular candidate is truly qualified. So if you're not a messianic ml rock star, sweating the degree might be a good idea. The beauty of a purely empirical craft is that results are the only arbiter of success. Anyone can get the right answer. It might be awhile before society catches up to and utilizes this powerful idea, but the hiring competitions on sites like kaggle are a great start [1]. Until then the best case is being the next Nate Silver. [1] http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extr...
- mathattack 13y ago
  Like anything, real world experience complements academic background, and over time the credential means less. But as a thought experiment... All things equal, who wouldn't hire someone with 2 years experience + a masters, over someone with just 2 years experience, or just the masters. I like to see something that signifies intelligence and curiosity, and then something that shows someone has done something with it. Usually this is a combination of academics and work. It can be shown just with work. It's hard to show it just with academics.
- wikiburner 13y ago
  The practice of statistical inference requires only a few key skills. Just curious, what do you think they are?
mattlutze 13y ago
“… Getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth,” Silver said in a Q&A with HBR’s Walter Frick. I'm in a data science class (hi carlob) right now. This is exactly how we've approached the subject. There were a few readings the first couple weeks to get people up to speed on Python (or scare them away if they didn't have any coding experience), then our homework problem sets have jumped straight into real-world data scraping and modeling. It's been a fantastic way to start learning the field, because after just 3.5-ish weeks I already feel like I have the tools I need to start exploring and fitting rudimentary models to the information I'm collecting. It's exciting and enabling, and I think that's definitely a strength of doing while learning.
kyllo 13y ago
I feel like Data Science is not a real academic field, and it's not something you need a college degree in. The programs are so new and rudimentary, the universities don't really even know what to teach the students in an "Intro to Data Science" class. But it's a buzzword in the media, and there's employer demand now, so I think that universities saw the $$$ and just ran with it. It's really more of a technical skillset and toolset for statisticians and other actual scientists who use statistics to test their hypotheses. Make Intro CS, Relational Algebra and Set Theory, and Computational Statistics electives for Bachelor of Science degrees. This really doesn't need its own degree program, and neither did "business intelligence" back when that was the buzzword for this. Universities aren't for "latest tools and fads" training.
- dragonwriter 13y ago
  > I feel like Data Science is not a real academic field, and it's not something you need a college degree in. Its an emergent academic field in the way that bioinformatics was a few years ago; "data science" is a name for something that's been taught in less breadth and depth as parts of other academic disciplines (including parts of the "research methods" curriculum in the life and social sciences, parts of statistics, parts of computer science), but certainly its as legitimate an area of focus as the disciplines its that parts of it have been hosted in. There's more than one way to divide up intellectual pursuit, and a number of ways that have cut across the previously predominant divisions have been coming to the fore recently. This is, really, a good thing.
  - kyllo 13y ago
    Yeah, I guess what I'm trying to say is that Data Science is really a "horizontal", not a "vertical" in academia, to borrow business jargon. It's an interdisciplinary research skill set that students and scholars across academic disciplines should be trained in. As such, I'm not convinced it needs its own degree programs, though.
    - dragonwriter 13y ago
      I think the quality of work being done in the disciplines you call "verticals" is improved by increasing the degree of development in the "horizontals" -- and that includes recognizing that those "horizontals" are legitimate fields of focus on their own, because the further development and focus on the "horizontal" feeds back into the tool sets available to all the "verticals" the "horizontal" touches.
mrcactu5 13y ago
most people don't really think logically or quantitatively. whatever math people learned in high school and college was forgotten when trying to make adult decisions. and they didn't even get to the good stuff. data science is successful now because they are applying quantitative methods to the social sciences -- problems most people think about. the average educated person can understand by using the magic of "machine learning" we can solve problems in politics, health, social issues, etc. Nate Silver's lack of a math background works to his advantage. he realizes the bar for quantitative understanding in the public sphere is very low. whatever little bit he does, he has to explain to a room full of drunk people in 30 seconds or less. he is very good at this. neither employers nor the public want to admit that at the end of the day SOMEONE has to be responsible for these calculations. i have a master's degree in math -- which nobody gave a hoot about (see item 5 in this article http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and-get-hired-as-one/ http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and... ) we weren't taught how to communicate our ideas and results to the public -- in fact we were discouraged from it. that is difficult to unlearn. i've heard of attempts to "democratize" machine learning. http://blog.bigml.com/2013/03/06/democratizing-machine-learning-with-c/ http://blog.bigml.com/2013/03/06/democratizing-machine-learn... articles like these are usually written by companies with a product or book to sell or distribute. what needs to be democratized is engineering based problem solving - using the powerful computers we hold in our pockets, our phones.
- slurry 13y ago
  Nate Silver's lack of a math background? Why does this keep getting repeated. He has a BA in economics from one of the most quantitative and rigorous programs in the country, plus some work at LSE. I'm sure he has more formal math and statistics education than most self-labeled "engineers" on HN.
  - geebee 13y ago
    Here's a link to the university of chicago's econ major requirements. http://collegecatalog.uchicago.edu/thecollege/economics/#programofstudy http://collegecatalog.uchicago.edu/thecollege/economics/#pro... Looks fairly rigorous. Can't say for sure, but it does appear that while you are encouraged to take the most rigorous calc sequence (typically required of math, hard science, engineering majors), econ majors are allowed to take an easier calculus track. That's pretty typical of econ undergraduate degrees - many universities do provide a slower, easier calculus track for certain majors (typically life science majors or social science majors). I have no idea what track Nate Silver took, and of course I can't speak to the "self-labeled" engineer. I'd agree that an econ from a reputable school has some mathematical rigor, but not quite at the level required of a typical math, hard science, or engineering major. Either way, I would agree with you that it is silly to say Nate Silver doesn't have a math background - clearly he does.
abbazabba 13y ago
Is there anyone around that "does data science" for a living that can talk about what it is they do exactly? I'm under the impression that "data scientist" is as ambiguous a term as "designer". What's the minimum amount of math or statistics required to be labeled a data scientist? And what's the minimum amount of programming too?

39 points by cdl 13y ago

Original source: http://blogs.hbr.org/2013/09/nate-silver-on-finding-a-mentor-teaching-yourself-statistics-and-not-settling-in-your-career/ http://blogs.hbr.org/2013/09/nate-silver-on-finding-a-mentor...

cruise02 13y ago
Is it just me, or is the gigaom article basically blogspam?
- jacques_chester 13y ago
  Saying that something is blogspam can be surprisingly controversial hereabouts. I find it safer to passive aggressively link to the original source.
  - cruise02 13y ago
    Sage advice. It's more productive to find the link and post it than it is to just complain about it, so I guess I should follow your lead. :)

clubhi 13y ago

This is the same with just about everything. When I started learning Clojure I forced myself to stop reading.

carlob 13y ago

There is something that somewhat irks me about the whole domain of data science. I still can't quite put my finger on it, but I think it's a mixture of lack of rigor, handwaving arguments. Maybe if it were called data journalism, or data storytelling rather than science I would somewhat lower my expectations and enjoy it more fully, but in my experience doing science and doing data science are not really close.

jonahx 13y ago
I think you could make the same argument about computer science, especially relating it to the actual practice of professional programming, which resembles pop culture more than science. And while both fields have mathematical underpinnings, the foundations of data science -- probability and statistics -- have deeper roots and a longer history. I guess I'm saying you shouldn't impugn the field itself because of its popular practice.
- carlob 13y ago
  I don't think what most programmers do is computer science, more like software engineering.
  - dllthomas 13y ago
    "more like", yes...
ig1 13y ago
Data science is about building testable models, which fundamentally is what science is about. What probably makes you uncomfortable is that Data Science (unlike most scientific fields) doesn't have a requirement for the models to make intuitive sense to humans.

pvnick 13y ago

I agree whole heartedly. When I first started playing around with data, despite my lite statistics background, I had no idea what I was doing and no amount of education would have prepared me for noisey, real life data sets. I had to become proficient with SQL and a certain type of problem solving and hacker mentality before I was able to do anything useful. What's interesting now though is that since returning to college and taking more advanced courses in probability and statistics, I now have a "practical" background that allow me to see real life use cases for almost everything I'm learning (oh wonderful Bayes theorem!) that is a huge advantage.

kartikkumar 13y ago

I find the don't-go-to-school-just-do-it mentality that pervades tech blogs, article, discussions a bit short-sighted. I fully appreciate the importance of getting "hands-on experience" in addition to book learning, but what I don't get is the black-and-white perspective that formal education gives you NO EXPERIENCE WHATSOEVER. As a PhD candidate, my perspective is quite definitely skewed, but my experience in school so far has not given me the impression that all I've been doing is book learning with no transferrable, real-world skills to work with data. I've had to work with plenty of data sets during my Masters and PhD research. I firmly believe that the experience I've gained from applying what I've learned in the classroom and from surveying existing literature on data mining and statistical methods stands me in good stead to tackle "real data" (whatever that means). Maybe I've got the wrong end of the stick, but I think that something more than the "odd stats class" can be of value to a budding data scientist ...

ThomPete 13y ago
Don't you find it odd that education start with the abstract to have you end up in reality. Data science is not about well structured data, it's not about learning what others do. I It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data. First experiment, learn to play around, explore, then apply theory.
- kartikkumar 13y ago
  Actually, I don't think the process of going from abstract to reality is that odd. The reason I say that is because I think understanding the abstract prepares you to be flexible. In terms of data science, the way I see this is that by experimenting without having a grounding in the algorithms, technologies etc. means that you can only really ever get good at the specific implementations you're working with. The abstract understanding is precisely what allows you to be versatile in adopting new technologies and seeing the implementation for what it is: one concrete version of the underlying abstract ideas. There's nothing that says that formal education only prepares you to work with well-structured data. In fact, I think that in grad school you are challenged to work at the cutting edge almost all the time. Granted, a lot of classroom examples at the undergrad level are perhaps trivial, but my experience is that grad school doesn't by definition only serve to prep you with pointless examples. > It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data. I totally agree, and I believe that school is not at odds with this. I'm working in the fields of planetary science and space engineering at the moment, and I am constantly stimulated to experiment with data, be creative and understand correlations. Maybe for some reason my experience is just not a good 3-sigma example ...

hooande 13y ago

It's worth pointing out that Nate Silver is a rare individual. He's been wildly successful with data science despite not having a deep academic background, which is not the common case. His story is a useful data point though. Do you need multiple degrees to master data science? Hell no. The practice of statistical inference requires only a few key skills. The important part is understanding how to think in terms of probability and algorithms, which doesn't require a rigorous understanding of proofs and formulae. An advanced academic degree is useful but not required. The world is looking for people who can consistently get the right answer, regardless of their background. The other side to this is that a degree will be required to work for wall st, google, harvard, etc. Degrees still matter and social signalling and so on and so forth. We all can't be as well known and popular as Nate Silver and at some point institutions need a way of establishing that a particular candidate is truly qualified. So if you're not a messianic ml rock star, sweating the degree might be a good idea. The beauty of a purely empirical craft is that results are the only arbiter of success. Anyone can get the right answer. It might be awhile before society catches up to and utilizes this powerful idea, but the hiring competitions on sites like kaggle are a great start [1]. Until then the best case is being the next Nate Silver. [1] http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extr...

mathattack 13y ago
Like anything, real world experience complements academic background, and over time the credential means less. But as a thought experiment... All things equal, who wouldn't hire someone with 2 years experience + a masters, over someone with just 2 years experience, or just the masters. I like to see something that signifies intelligence and curiosity, and then something that shows someone has done something with it. Usually this is a combination of academics and work. It can be shown just with work. It's hard to show it just with academics.
wikiburner 13y ago
The practice of statistical inference requires only a few key skills. Just curious, what do you think they are?

mattlutze 13y ago

“… Getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth,” Silver said in a Q&A with HBR’s Walter Frick. I'm in a data science class (hi carlob) right now. This is exactly how we've approached the subject. There were a few readings the first couple weeks to get people up to speed on Python (or scare them away if they didn't have any coding experience), then our homework problem sets have jumped straight into real-world data scraping and modeling. It's been a fantastic way to start learning the field, because after just 3.5-ish weeks I already feel like I have the tools I need to start exploring and fitting rudimentary models to the information I'm collecting. It's exciting and enabling, and I think that's definitely a strength of doing while learning.

kyllo 13y ago

I feel like Data Science is not a real academic field, and it's not something you need a college degree in. The programs are so new and rudimentary, the universities don't really even know what to teach the students in an "Intro to Data Science" class. But it's a buzzword in the media, and there's employer demand now, so I think that universities saw the $$$ and just ran with it. It's really more of a technical skillset and toolset for statisticians and other actual scientists who use statistics to test their hypotheses. Make Intro CS, Relational Algebra and Set Theory, and Computational Statistics electives for Bachelor of Science degrees. This really doesn't need its own degree program, and neither did "business intelligence" back when that was the buzzword for this. Universities aren't for "latest tools and fads" training.

dragonwriter 13y ago
> I feel like Data Science is not a real academic field, and it's not something you need a college degree in. Its an emergent academic field in the way that bioinformatics was a few years ago; "data science" is a name for something that's been taught in less breadth and depth as parts of other academic disciplines (including parts of the "research methods" curriculum in the life and social sciences, parts of statistics, parts of computer science), but certainly its as legitimate an area of focus as the disciplines its that parts of it have been hosted in. There's more than one way to divide up intellectual pursuit, and a number of ways that have cut across the previously predominant divisions have been coming to the fore recently. This is, really, a good thing.
- kyllo 13y ago
  Yeah, I guess what I'm trying to say is that Data Science is really a "horizontal", not a "vertical" in academia, to borrow business jargon. It's an interdisciplinary research skill set that students and scholars across academic disciplines should be trained in. As such, I'm not convinced it needs its own degree programs, though.
  - dragonwriter 13y ago
    I think the quality of work being done in the disciplines you call "verticals" is improved by increasing the degree of development in the "horizontals" -- and that includes recognizing that those "horizontals" are legitimate fields of focus on their own, because the further development and focus on the "horizontal" feeds back into the tool sets available to all the "verticals" the "horizontal" touches.

mrcactu5 13y ago

most people don't really think logically or quantitatively. whatever math people learned in high school and college was forgotten when trying to make adult decisions. and they didn't even get to the good stuff. data science is successful now because they are applying quantitative methods to the social sciences -- problems most people think about. the average educated person can understand by using the magic of "machine learning" we can solve problems in politics, health, social issues, etc. Nate Silver's lack of a math background works to his advantage. he realizes the bar for quantitative understanding in the public sphere is very low. whatever little bit he does, he has to explain to a room full of drunk people in 30 seconds or less. he is very good at this. neither employers nor the public want to admit that at the end of the day SOMEONE has to be responsible for these calculations. i have a master's degree in math -- which nobody gave a hoot about (see item 5 in this article http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and-get-hired-as-one/ http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and... ) we weren't taught how to communicate our ideas and results to the public -- in fact we were discouraged from it. that is difficult to unlearn. i've heard of attempts to "democratize" machine learning. http://blog.bigml.com/2013/03/06/democratizing-machine-learning-with-c/ http://blog.bigml.com/2013/03/06/democratizing-machine-learn... articles like these are usually written by companies with a product or book to sell or distribute. what needs to be democratized is engineering based problem solving - using the powerful computers we hold in our pockets, our phones.

slurry 13y ago
Nate Silver's lack of a math background? Why does this keep getting repeated. He has a BA in economics from one of the most quantitative and rigorous programs in the country, plus some work at LSE. I'm sure he has more formal math and statistics education than most self-labeled "engineers" on HN.
- geebee 13y ago
  Here's a link to the university of chicago's econ major requirements. http://collegecatalog.uchicago.edu/thecollege/economics/#programofstudy http://collegecatalog.uchicago.edu/thecollege/economics/#pro... Looks fairly rigorous. Can't say for sure, but it does appear that while you are encouraged to take the most rigorous calc sequence (typically required of math, hard science, engineering majors), econ majors are allowed to take an easier calculus track. That's pretty typical of econ undergraduate degrees - many universities do provide a slower, easier calculus track for certain majors (typically life science majors or social science majors). I have no idea what track Nate Silver took, and of course I can't speak to the "self-labeled" engineer. I'd agree that an econ from a reputable school has some mathematical rigor, but not quite at the level required of a typical math, hard science, or engineering major. Either way, I would agree with you that it is silly to say Nate Silver doesn't have a math background - clearly he does.

abbazabba 13y ago

Is there anyone around that "does data science" for a living that can talk about what it is they do exactly? I'm under the impression that "data scientist" is as ambiguous a term as "designer". What's the minimum amount of math or statistics required to be labeled a data scientist? And what's the minimum amount of programming too?