4 ms·

My view is from a small startup with little to no room for single purpose employees. When I first started hiring and working with data scientist my view was th

My view is from a small startup with little to no room for single purpose employees. When I first started hiring and working with data scientist my view was this: If you can only manipulate data and run it through pipelines to generate models then you can't do enough to be highly valuable. You either need to have a strong enough background in CS to build the pipelines / tools or a strong enough mathematics background to be able to propose cutting edge new ideas. From my experience it is hard to find someone who has one of these skill just from a University "data science" program. At a small company (at least ones that I have worked with) being only proficient in R and basic Python isn't enough. That being said, I have met and handful of Data Scientist who were very smart and self motivated enough to pick up on the lacking skills when given the chance. My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research?

proverbialbunny 6y ago
I would be cautious about that. I've worked in the startup space for over 10 years now as a data scientist, often the first one hired on, working on the pipes. From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relying on the data scientist to be profitable, I'd be cautious with these types. or 2) Someone who is senior, beyond senior really, who has worked both jobs, and doesn't mind doing both jobs. This unicorn is so rare it is mythical. The joke when the terminology was created is they're so rare no one has ever seen one, hence unicorn. Me, I can not do the work I need to do if I'm on call. That is where I draw the line. That means hiring someone to monitor the infrastructure. Furthermore, I'm an okay architect, but you really do want to hire a specialist if you can help it for that. Do I help them with the infrastructure? Absolutely, but they're on call if a server is on fire. They have the admin login credentials, not me. I get wearing multiple hats, but keep in mind to be a data scientist you're already wearing multiple hats. Being a data scientist is like double majoring and getting a phd. At what point are they stretched too thin? The consensus in the industry is they're already stretched too thin and should be broken up into different specialized roles. >My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research? That is the standard role, even at startups. However, the industry consensus these days is data scientists should have more responsibility when it comes to deploying models than previous standards.[1] So data scientists are being pushed in a more engineering direction, not with hosting sql servers and infrastructure, but with working with engineers to make sure the models are monitored properly. This change comes from model deployment being further automated as time goes on, making it easier for the data scientist to have more responsibility during this stage. [1] source: https://www.dominodatalab.com/static/gfx/uploads/domino-managing-ds.pdf https://www.dominodatalab.com/static/gfx/uploads/domino-mana... page 9. Suboptimal organization and incentive structures.
- a_zaydak 6y ago
  Thanks for the feedback! Seems like you and I both have had a bit of experience being first engineering hires at startups but have had very different experiences when it comes to rolls or a data scientist. I appreciate that.
  - proverbialbunny 6y ago
    Np. There is a common trend in the industry where a company hires on a data scientist, doesn't know the data prerequisites (specifically labeled data), the data scientist struggles, after a while the company fires the data scientist. This leaves the company with a bad taste in their mouth. In recent years I tend to get hired on as a specialist to help fix this. (And yes, I've been the first engineer hired on too.) What's interesting is they tend to struggle in two different ways: 1) The data scientist that is gung ho about infrastructure work, jumps in, and then ends up doing a bad job, because it's not their strength. They end up getting let go for not being ideal at that work. 2) The data scientist who struggles with the idea of infrastructure work at all, jumps into other roles they're good at like data analyst work, helps the company in that way, but ultimately because they did not push to get an infrastructure engineer hired, they end up let go as well. Me, I go out of my way to get an infrastructure engineer / data engineer hired early on. Also, I have worked as an engineer, so I tend to do a lot of the "hard" stuff most software engineers struggle with early on, if applicable. Eg, at one job I wrote a compression format to reduce battery drain on our devices that were collecting data. Most data scientists struggle when it comes to CS/engineering skills (4/5th of them), so it's not uncommon for them early one while the pipes are being built to do data analyst and BI work. BI work to automate reports, which management loves, and DA work to show some amazing future service the company might be able provide to its customers. It's selling the sun and the moon really, but it gets management inspired, and helps them know what data to collect. It's not unheard of to need a minimum of two years of collected data before building a model that can be deployed becomes feasible. This can be hard on the data scientist, because there is a lot of down time before that. Many get fired during this time even when they're doing a good job. They have to wear multiple hats, but it's analyst roles (like BI work). Technically a data scientist is a kind of analyst, not engineer, so it makes sense that wearing multiple hats for them tilts in the analyst direction, not the engineering direction. I've been writing code since I was 8 years old, so I'm one of the unusual ones that tilts in the engineering direction, but I think it is unreasonable to expect that from the average data scientist. Let them do what they do best, and hire someone else who can round everything out and you'll be in a good place. Unicorns aside, you'll need a minimum of two professionals for a data project to succeed.
jorpal 6y ago
There are certainly roles out there for a Data Scientist who just crunches numbers. A good friend of mine does exactly that for a large traditional retail corporation. Just by using standard ML tools he replaced a whole team of analysts for pricing items. Maybe not in cutting edge tech companies, but roles like that are all over the economy still.

proverbialbunny 6y ago

I would be cautious about that. I've worked in the startup space for over 10 years now as a data scientist, often the first one hired on, working on the pipes. From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relying on the data scientist to be profitable, I'd be cautious with these types. or 2) Someone who is senior, beyond senior really, who has worked both jobs, and doesn't mind doing both jobs. This unicorn is so rare it is mythical. The joke when the terminology was created is they're so rare no one has ever seen one, hence unicorn. Me, I can not do the work I need to do if I'm on call. That is where I draw the line. That means hiring someone to monitor the infrastructure. Furthermore, I'm an okay architect, but you really do want to hire a specialist if you can help it for that. Do I help them with the infrastructure? Absolutely, but they're on call if a server is on fire. They have the admin login credentials, not me. I get wearing multiple hats, but keep in mind to be a data scientist you're already wearing multiple hats. Being a data scientist is like double majoring and getting a phd. At what point are they stretched too thin? The consensus in the industry is they're already stretched too thin and should be broken up into different specialized roles. >My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research? That is the standard role, even at startups. However, the industry consensus these days is data scientists should have more responsibility when it comes to deploying models than previous standards.[1] So data scientists are being pushed in a more engineering direction, not with hosting sql servers and infrastructure, but with working with engineers to make sure the models are monitored properly. This change comes from model deployment being further automated as time goes on, making it easier for the data scientist to have more responsibility during this stage. [1] source: https://www.dominodatalab.com/static/gfx/uploads/domino-managing-ds.pdf https://www.dominodatalab.com/static/gfx/uploads/domino-mana... page 9. Suboptimal organization and incentive structures.

a_zaydak 6y ago
Thanks for the feedback! Seems like you and I both have had a bit of experience being first engineering hires at startups but have had very different experiences when it comes to rolls or a data scientist. I appreciate that.
- proverbialbunny 6y ago
  Np. There is a common trend in the industry where a company hires on a data scientist, doesn't know the data prerequisites (specifically labeled data), the data scientist struggles, after a while the company fires the data scientist. This leaves the company with a bad taste in their mouth. In recent years I tend to get hired on as a specialist to help fix this. (And yes, I've been the first engineer hired on too.) What's interesting is they tend to struggle in two different ways: 1) The data scientist that is gung ho about infrastructure work, jumps in, and then ends up doing a bad job, because it's not their strength. They end up getting let go for not being ideal at that work. 2) The data scientist who struggles with the idea of infrastructure work at all, jumps into other roles they're good at like data analyst work, helps the company in that way, but ultimately because they did not push to get an infrastructure engineer hired, they end up let go as well. Me, I go out of my way to get an infrastructure engineer / data engineer hired early on. Also, I have worked as an engineer, so I tend to do a lot of the "hard" stuff most software engineers struggle with early on, if applicable. Eg, at one job I wrote a compression format to reduce battery drain on our devices that were collecting data. Most data scientists struggle when it comes to CS/engineering skills (4/5th of them), so it's not uncommon for them early one while the pipes are being built to do data analyst and BI work. BI work to automate reports, which management loves, and DA work to show some amazing future service the company might be able provide to its customers. It's selling the sun and the moon really, but it gets management inspired, and helps them know what data to collect. It's not unheard of to need a minimum of two years of collected data before building a model that can be deployed becomes feasible. This can be hard on the data scientist, because there is a lot of down time before that. Many get fired during this time even when they're doing a good job. They have to wear multiple hats, but it's analyst roles (like BI work). Technically a data scientist is a kind of analyst, not engineer, so it makes sense that wearing multiple hats for them tilts in the analyst direction, not the engineering direction. I've been writing code since I was 8 years old, so I'm one of the unusual ones that tilts in the engineering direction, but I think it is unreasonable to expect that from the average data scientist. Let them do what they do best, and hire someone else who can round everything out and you'll be in a good place. Unicorns aside, you'll need a minimum of two professionals for a data project to succeed.

jorpal 6y ago

There are certainly roles out there for a Data Scientist who just crunches numbers. A good friend of mine does exactly that for a large traditional retail corporation. Just by using standard ML tools he replaced a whole team of analysts for pricing items. Maybe not in cutting edge tech companies, but roles like that are all over the economy still.