4 ms·

I always liked data crunching, databases and data topics in general. Also, most of the software development knowledge I have I didn't learn in the official scho

I always liked data crunching, databases and data topics in general. Also, most of the software development knowledge I have I didn't learn in the official school curriculum, but rather from books, online courses and real-world experience. Now, having that in mind, how realistic is for a guy in mid-30s, a very good software developer (enthusiastic about functional programming, if that matters) to pick up enough knowledge about data science to be able to actually take the data-scientist role in some company? Edit: Rephrase the question.

aangjie 10y ago
Depends on the stage the company is at... I've a similar background, but took up a Data Scientist role. However and here's the rub. The company has very little data-pipeline setup and functioning , that much of the work is laying out the plumbing, ensuring it works without getting blocked etc.. I've tried my hand at a few predictive models and cross-validation accuracy looks good, but can't test them on real-life scenario, as some of the data-source it depends on is not coming in. What's worse is they don't have a capable and dedicated devops, so it's the CTO who had started this data puller project and hasn't delegated/handed-over either. So I'm reduced sitting on my ass, warming the seat/chair.
selectron 10y ago
You could absolutely learn all the skills and knowledge to be a good data scientist, the real question is if you could get hired. I have been looking into data science and I think the demand for data scientists is not as large as it seems, but it is growing. There are very few junior data science positions. Especially without a PhD or an inside contact it could be challenging to get your foot in the door. My advice would be to check out Kaggle, try out a problem and see if you enjoy it.
barry-cotter 10y ago
There are multiple Data Science boot camps. The fact that these places exist and claim to be able to get people jobs after three months learning suggests you could do it. Look at Signal Data Science.
jllopezpino 10y ago
I agree with the previous messages. There are basically two paths: A) Learn from others. There are some good programs like Data Science Retreat. B) Self-learn and use Kaggle competitions to prove it.
IanCal 10y ago
Totally realistic. It'll depend on what the company is, and what they're looking for (and hopefully, this will match what they should be looking for). While people can often focus on applying the latest deep learning thought-vector approach to their BIG DATA, there's an enormous gulf between the common condition of data and this aspiration. You don't need PhD level stats and machine learning to apply the things that many companies actually could benefit from. Storing, maintaining and managing the data properly is a start. Then working with the people in the business to get insights from the data they have. Often simple aggregations and visualisations can provide enormous benefit. Being able to show correlations, and sometimes even just being able to show how noisy things are can be important. Big questions: How is reality different from what we think? How are these differences important for the company? That might be a correlation that we don't expect, or a lack of one we think is there. Part of a next step might be to become more "pro-active" and design new experiments to answer questions that can't quite be answered yet due to a lack of data. Beyond that you're heading towards bringing a new feature into a product. Does your current company have some data? Do they spend a lot of time emailing spreadsheets around, or have salesforce or a proper database? See if you can make something useful for your work that's based on an analysis of that data (and possibly do some work to smooth those workflows of passing data around).
- mswen 10y ago
  IanCal - I upvoted you but let me comment for extra emphasis. I remember coming out of PhD level quantitative social science studies in academia back in the late 90s where I was using K-means clustering, Factor analysis, multiple linear regression, ANOVA and more. When I moved into marketing research and dealing with company data it was shocking and disheartening how little of those skills I could actually deploy. Data quality, data management and just the cost of capturing relevant data was so high that we were reduced to much simpler analyses. Over time I came to appreciate exactly what you were saying. Many companies can be helped by fairly simple analyses. Fast forward - data is getting much cheaper and now all these years later my more advanced stats skills seem to actually matter. But even in this apparent abundance ... data management and data quality assurance is often lacking or significantly underfunded and simple aggregations and analyses still make the most difference in many organizations.
kriro 10y ago
To take a page from the authors playbook...create some artifacts. Find interesting data sets, do interesting stuff with said data sets and write about it :)
mathattack 10y ago
Absolutely realistic, though it helps to have good intuition about math. I've seen smart people without data science experience make the leap after a 12 week boot camp, or finishing the Johns Hopkins Coursera certificate. If you're good at picking stuff up online, the latter is probably the best start. There is such a shortage of data scientists that most companies will settle for 2 out of 3 of the trinity of Stats, Programming and Domain Knowledge. If the candidate is smart enough, sometimes 1 out of 3 is enough.
JasonPunyon 10y ago
Hey, I'm the Jason Punyon drob's gushing about in the post. I wanted to tell you that this is 100% totally realistic, everyone on the Data team at Stack Overflow besides Dave Robinson is living proof. In the beginning the Data Science Team at Stack Overflow was just me and Kevin Montrose. We're both ~30, we had some skills that were data-ish, but they were general critical thinking and math skills. By no means were we statisticians or data scientists by trade like Dave Robinson. However, the data at Stack Overflow was very amenable to analysis, and we were able to ship a personalized machine learning product after a year of work. That year was plagued by (my, mostly) engineering failures (http://jasonpunyon.com/blog/2015/02/12/providence-failure-is-always-an-option/ http://jasonpunyon.com/blog/2015/02/12/providence-failure-is...) and a bit of unfortunate lack-of-empiricism, not "data science" problems. And post launch of Providence, the engineering remains the bigger issue. It's just hard to get this shit right. Our record on getting experiments right the first time is spotty, and at Stack Overflow levels of traffic you might be dumping a week or more of time on a single screwup. We've had to rerun many experiments because of various things going wrong with the code (screwed up the A/B testing code, not counting the right things, visual issues with the ads we didn't catch during design, etc etc etc). There's a litany of things you've gotta get right engineering wise before data even comes into the picture. So if you're gonna take the plunge, know that it's 100% possible but 100% difficult, and maybe not for the reasons you think.
brey 10y ago
I hire plenty of data scientists - if I had to give a single piece of advice: gain a deep understanding of the techniques you're using. it's not sufficient to say "oh, we used a support vector machine", and when prompted for more detail about how it works to shrug and say you just copied and pasted code from the internet and tweaked parameters until it worked.
muraiki 10y ago
I'm also a self taught programmer. I recently started using https://www.dataquest.io/ https://www.dataquest.io/ (not affiliated with them) and have found it to be a great data science resource for people who prefer a hands-on approach to learning. Previously I'd tried various MOOCs but the lecture format with only a few questions to work on was not helpful for me.

aangjie 10y ago

Depends on the stage the company is at... I've a similar background, but took up a Data Scientist role. However and here's the rub. The company has very little data-pipeline setup and functioning , that much of the work is laying out the plumbing, ensuring it works without getting blocked etc.. I've tried my hand at a few predictive models and cross-validation accuracy looks good, but can't test them on real-life scenario, as some of the data-source it depends on is not coming in. What's worse is they don't have a capable and dedicated devops, so it's the CTO who had started this data puller project and hasn't delegated/handed-over either. So I'm reduced sitting on my ass, warming the seat/chair.

selectron 10y ago

You could absolutely learn all the skills and knowledge to be a good data scientist, the real question is if you could get hired. I have been looking into data science and I think the demand for data scientists is not as large as it seems, but it is growing. There are very few junior data science positions. Especially without a PhD or an inside contact it could be challenging to get your foot in the door. My advice would be to check out Kaggle, try out a problem and see if you enjoy it.

barry-cotter 10y ago

There are multiple Data Science boot camps. The fact that these places exist and claim to be able to get people jobs after three months learning suggests you could do it. Look at Signal Data Science.

jllopezpino 10y ago

I agree with the previous messages. There are basically two paths: A) Learn from others. There are some good programs like Data Science Retreat. B) Self-learn and use Kaggle competitions to prove it.

IanCal 10y ago

Totally realistic. It'll depend on what the company is, and what they're looking for (and hopefully, this will match what they should be looking for). While people can often focus on applying the latest deep learning thought-vector approach to their BIG DATA, there's an enormous gulf between the common condition of data and this aspiration. You don't need PhD level stats and machine learning to apply the things that many companies actually could benefit from. Storing, maintaining and managing the data properly is a start. Then working with the people in the business to get insights from the data they have. Often simple aggregations and visualisations can provide enormous benefit. Being able to show correlations, and sometimes even just being able to show how noisy things are can be important. Big questions: How is reality different from what we think? How are these differences important for the company? That might be a correlation that we don't expect, or a lack of one we think is there. Part of a next step might be to become more "pro-active" and design new experiments to answer questions that can't quite be answered yet due to a lack of data. Beyond that you're heading towards bringing a new feature into a product. Does your current company have some data? Do they spend a lot of time emailing spreadsheets around, or have salesforce or a proper database? See if you can make something useful for your work that's based on an analysis of that data (and possibly do some work to smooth those workflows of passing data around).

mswen 10y ago
IanCal - I upvoted you but let me comment for extra emphasis. I remember coming out of PhD level quantitative social science studies in academia back in the late 90s where I was using K-means clustering, Factor analysis, multiple linear regression, ANOVA and more. When I moved into marketing research and dealing with company data it was shocking and disheartening how little of those skills I could actually deploy. Data quality, data management and just the cost of capturing relevant data was so high that we were reduced to much simpler analyses. Over time I came to appreciate exactly what you were saying. Many companies can be helped by fairly simple analyses. Fast forward - data is getting much cheaper and now all these years later my more advanced stats skills seem to actually matter. But even in this apparent abundance ... data management and data quality assurance is often lacking or significantly underfunded and simple aggregations and analyses still make the most difference in many organizations.

kriro 10y ago

To take a page from the authors playbook...create some artifacts. Find interesting data sets, do interesting stuff with said data sets and write about it :)

mathattack 10y ago

Absolutely realistic, though it helps to have good intuition about math. I've seen smart people without data science experience make the leap after a 12 week boot camp, or finishing the Johns Hopkins Coursera certificate. If you're good at picking stuff up online, the latter is probably the best start. There is such a shortage of data scientists that most companies will settle for 2 out of 3 of the trinity of Stats, Programming and Domain Knowledge. If the candidate is smart enough, sometimes 1 out of 3 is enough.

JasonPunyon 10y ago

Hey, I'm the Jason Punyon drob's gushing about in the post. I wanted to tell you that this is 100% totally realistic, everyone on the Data team at Stack Overflow besides Dave Robinson is living proof. In the beginning the Data Science Team at Stack Overflow was just me and Kevin Montrose. We're both ~30, we had some skills that were data-ish, but they were general critical thinking and math skills. By no means were we statisticians or data scientists by trade like Dave Robinson. However, the data at Stack Overflow was very amenable to analysis, and we were able to ship a personalized machine learning product after a year of work. That year was plagued by (my, mostly) engineering failures (http://jasonpunyon.com/blog/2015/02/12/providence-failure-is-always-an-option/ http://jasonpunyon.com/blog/2015/02/12/providence-failure-is...) and a bit of unfortunate lack-of-empiricism, not "data science" problems. And post launch of Providence, the engineering remains the bigger issue. It's just hard to get this shit right. Our record on getting experiments right the first time is spotty, and at Stack Overflow levels of traffic you might be dumping a week or more of time on a single screwup. We've had to rerun many experiments because of various things going wrong with the code (screwed up the A/B testing code, not counting the right things, visual issues with the ads we didn't catch during design, etc etc etc). There's a litany of things you've gotta get right engineering wise before data even comes into the picture. So if you're gonna take the plunge, know that it's 100% possible but 100% difficult, and maybe not for the reasons you think.

brey 10y ago

I hire plenty of data scientists - if I had to give a single piece of advice: gain a deep understanding of the techniques you're using. it's not sufficient to say "oh, we used a support vector machine", and when prompted for more detail about how it works to shrug and say you just copied and pasted code from the internet and tweaked parameters until it worked.

muraiki 10y ago

I'm also a self taught programmer. I recently started using https://www.dataquest.io/ https://www.dataquest.io/ (not affiliated with them) and have found it to be a great data science resource for people who prefer a hands-on approach to learning. Previously I'd tried various MOOCs but the lecture format with only a few questions to work on was not helpful for me.