4 ms·

I always felt that tech-focused data scientists should also be required to know how process data end-to-end; at minimum, from a SQL database to deployed model,

I always felt that tech-focused data scientists should also be required to know how process data end-to-end; at minimum, from a SQL database to deployed model, but knowing how to collect & clean data is important too. It seems like the industry is trying fill the gap that was created by a glut of people without math/cs backgrounds going into 5-week data science courses who then need hand-holding when they get real jobs. Data science & engineering should be treated as a single collection of skill-sets. Lacking ETL experience is a major deficit, considering how prevalent that kind of work is. This might just be my personal biases coming through. I consider myself a "full-stack" data scientist & engineer. But because data scientists who can work on the backends are rare, I always end up doing the plumbing while other people do the fun analysis work. I think companies that are data "science" heavy are going to be at huge disadvantage soon. Tools like Rekognition and Google AI APIs are making the model training & deployment aspect almost trivial. At some point, the only real work involved in this space will be the data "engineering."

superbcarrot 6y ago
> Data science & engineering should be treated as a single collection of skill-sets. This can be tough because there could be a lot in that skill set. You can't realistically expect someone to have solid knowledge of statistics including specialising in the sub-field and type of algorithms that your product needs, and also be able to write good code and act as a developer, and also have solid knowledge of all the tools for data streaming/processing/ETL. There is a point at which you're just stretching yourself too thin if you try to do all of these at once. Of course, stuff like knowing how to interact with a database or employing good software development practices should be a very basic prerequisite and some scientists certainly shift things too far in the other direction and use their academic knowledge as an excuse to write poor code and not learn new tools. I guess what I'm trying to say is that they are distinct skills but you still need all of them to some extent and striking the correct balance in one's skillset is really difficult.
- mywittyname 6y ago
  These are all skills taught in standard computer science programs. Granted, some are electives, like high-level stats. But even back in 2010, data science electives were available to fill the gaps. I took three DS&E classes in college with projects that were end-to-end platforms, where you'd have to collect, clean, and analyze the data, then build, test, and deploy models from it. I would certainly hope that college courses are even more comprehensive after 10 years and an explosion in interest for the field. Also, much like being a full stack developer, a full stack data engineer doesn't need to know everything at a master level. But that you can at least handle tasks at most points in the chain.

superbcarrot 6y ago

> Data science & engineering should be treated as a single collection of skill-sets. This can be tough because there could be a lot in that skill set. You can't realistically expect someone to have solid knowledge of statistics including specialising in the sub-field and type of algorithms that your product needs, and also be able to write good code and act as a developer, and also have solid knowledge of all the tools for data streaming/processing/ETL. There is a point at which you're just stretching yourself too thin if you try to do all of these at once. Of course, stuff like knowing how to interact with a database or employing good software development practices should be a very basic prerequisite and some scientists certainly shift things too far in the other direction and use their academic knowledge as an excuse to write poor code and not learn new tools. I guess what I'm trying to say is that they are distinct skills but you still need all of them to some extent and striking the correct balance in one's skillset is really difficult.

mywittyname 6y ago
These are all skills taught in standard computer science programs. Granted, some are electives, like high-level stats. But even back in 2010, data science electives were available to fill the gaps. I took three DS&E classes in college with projects that were end-to-end platforms, where you'd have to collect, clean, and analyze the data, then build, test, and deploy models from it. I would certainly hope that college courses are even more comprehensive after 10 years and an explosion in interest for the field. Also, much like being a full stack developer, a full stack data engineer doesn't need to know everything at a master level. But that you can at least handle tasks at most points in the chain.