3 ms·

In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in stati

In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in statistics and algorithms to get anything of any value done. The fact that you think it takes a PhD to be a decent data scientist indicates that you're out of your depth on this one. You don't need a PhD to get useful work done in these fields. You need to work hard and tackle difficult math. It takes years, but it can be done if you have the talent and drive. A prestigious (top-10) PhD certainly makes your life easier in getting the top jobs, but it doesn't really make you more (or less) able to fulfill them. I don't have a PhD and can do what 95+ percent of "PhD Data Scientists" do for work. PhD is (a) focused work on a specific, usually narrow problem, and (b) years of self-study that required to know enough to attack said problem. For real-world data science, (a) only matters in the ~1% chance of overlap between your dissertation and the needs of your employer, and (b) doesn't require five years in an academic institution (although it probably does require about that much time, if you study on your own, since you're likely to be doing much of the work on your own time). The PhD is a valuable experience and I don't mean to denigrate it. I often wish I had gotten one, in my 20s, instead of becoming a world-class expert on software office politics and "only" an intermediate-plus Haskell/Clojure/machine-learning guy. The PhD is a great experience for many people, but I don't think it belongs on a pedestal.

Daishiman 12y ago
I'm not talking strictly about a PhD, but about PhD level work, some of which can't really be done on your own unless you're a supremely talented individual. You can look at some of the modern ML algorithms and see what I mean; many people that I know have worked with Latent Dirichlet Allocation, but they have no idea how the model works, and there's no way they could extend to work online or under certain performance or storage constraints without havin spent months and years working on that problem. That's not a realistic expectation for anyone in the field. Yes, the algorithms you find in Weka and other ML toolkits are useful, but the actual "Big Data" problems have their own performance and algorithmic constraints that are far, far beyond dedicated self-learners.

Daishiman 12y ago

I'm not talking strictly about a PhD, but about PhD level work, some of which can't really be done on your own unless you're a supremely talented individual. You can look at some of the modern ML algorithms and see what I mean; many people that I know have worked with Latent Dirichlet Allocation, but they have no idea how the model works, and there's no way they could extend to work online or under certain performance or storage constraints without havin spent months and years working on that problem. That's not a realistic expectation for anyone in the field. Yes, the algorithms you find in Weka and other ML toolkits are useful, but the actual "Big Data" problems have their own performance and algorithmic constraints that are far, far beyond dedicated self-learners.