3 ms·

I also hire data scientists & have a similar experience. As far as I can tell, many people are taught to start from a statistical/machine learning method and ap

I also hire data scientists & have a similar experience. As far as I can tell, many people are taught to start from a statistical/machine learning method and apply it to a problem, but very few are taught to start from a problem and figure what techniques to use. Honestly, 95% of the time I solve my questions through iterative SQL queries in a few hours, while I see most people using laborious statistical methods the first chance they get.

nickdavidhaynes 10y ago
Call it "the kaggle effect" - once someone defines the problem and the metric you'll be graded on and gives you a relatively clean dataset, "solving the problem" is just as simple as importing xgboost and plowing away. But there is often an under-appreciation among people without much job experience how hard it is to get to that point. The OP article touched on it a bit, but really, the most difficult job a data scientist has is defining what problem they're trying to solve and getting buy-in from other business stakeholders. And frankly, no data science masters program or boot camp can teach those skills.
cosmie 10y ago
> 95% of the time I solve my questions through iterative SQL queries in a few hours, while I see most people using laborious statistical methods the first chance they get. The issue seems to be a mismatch between your posting and your workload. I do hiring for a data team, and explicitly don't advertise a data science role. While we do have projects that are advanced enough to fall under a data science moniker, the majority of candidates we got for that role had very... academic expectations. But a business isn't a static, cleanroom environment with everything already collected, cleaned, standardized, validated, and normalized for use. Re-titling the job posting to Data Specialist or Data Analyst resulted in a lot more candidates that are perfectly well suited to the type of problem solving you mentioned. There's an endless number of business problems where this skillset can be applied, making them very flexible and providing high labor utilization. Including getting to a "good enough" state for the few problems we have that could benefit from the more advanced statistical methods a data science candidate would bring to the table.
- SatvikBeri 10y ago
  Yeah–to be clear, I pretty much totally revamped the hiring process once I became a manager, and was speaking mostly from previous experience. I've found splitting up the job into different titles "Data Analyst", "Data Scientist", and "Data Engineer" depending on the actual role to work pretty well. That said, even with the vast majority of analyst candidates, I find them very eager to apply known methods–flexibility and problem-first thinking is rare and extremely valuable.
  - daveguy 10y ago
    Those titles. What type of roles do they cover? Is there a quick summary -- particularly between analyst and scientist. I expect engineer is source quality, repeatability, accuracy, precision, feature engineering, etc. In other words making the data stable and easily consumed, whether that is directly from the instrument or the charts for the final decision. The nuance between analyst and scientist is less clear. Can you describe what type of candidates the two draws or what you look for depending on the title?
    - bigger_cheese 10y ago
      My job title is currently "Data Engineer" I work in an industrial plant. Here's my two cents: My background is in Engineering (I'm a materials engineer by qualification). What differentiates me from a statistician, analyst etc is my domain knowledge. I have almost 15 years experience working with industrial processes. I have the background knowledge of chemistry, thermodynamics, mechanics etc. Which someone with a stats background would be lacking. So when I am asked to optimize an industrial process I can utilize that expertise whilst developing models. I would expect that a data scientist would know more about machine learning and would have a much stronger stats background than me. They'd also probably write much better code (I work in C/C++ and SAS, from what I have seen data scientists tend to be Python/R focused).
    - l_t 10y ago
      Not the OP, but in my experience a "Data analyst" is mostly responsible for writing analytical SQL queries and generating reports. So they don't require a strong math background or programming skills (other than SQL).

nickdavidhaynes 10y ago

Call it "the kaggle effect" - once someone defines the problem and the metric you'll be graded on and gives you a relatively clean dataset, "solving the problem" is just as simple as importing xgboost and plowing away. But there is often an under-appreciation among people without much job experience how hard it is to get to that point. The OP article touched on it a bit, but really, the most difficult job a data scientist has is defining what problem they're trying to solve and getting buy-in from other business stakeholders. And frankly, no data science masters program or boot camp can teach those skills.

cosmie 10y ago

> 95% of the time I solve my questions through iterative SQL queries in a few hours, while I see most people using laborious statistical methods the first chance they get. The issue seems to be a mismatch between your posting and your workload. I do hiring for a data team, and explicitly don't advertise a data science role. While we do have projects that are advanced enough to fall under a data science moniker, the majority of candidates we got for that role had very... academic expectations. But a business isn't a static, cleanroom environment with everything already collected, cleaned, standardized, validated, and normalized for use. Re-titling the job posting to Data Specialist or Data Analyst resulted in a lot more candidates that are perfectly well suited to the type of problem solving you mentioned. There's an endless number of business problems where this skillset can be applied, making them very flexible and providing high labor utilization. Including getting to a "good enough" state for the few problems we have that could benefit from the more advanced statistical methods a data science candidate would bring to the table.

SatvikBeri 10y ago
Yeah–to be clear, I pretty much totally revamped the hiring process once I became a manager, and was speaking mostly from previous experience. I've found splitting up the job into different titles "Data Analyst", "Data Scientist", and "Data Engineer" depending on the actual role to work pretty well. That said, even with the vast majority of analyst candidates, I find them very eager to apply known methods–flexibility and problem-first thinking is rare and extremely valuable.
- daveguy 10y ago
  Those titles. What type of roles do they cover? Is there a quick summary -- particularly between analyst and scientist. I expect engineer is source quality, repeatability, accuracy, precision, feature engineering, etc. In other words making the data stable and easily consumed, whether that is directly from the instrument or the charts for the final decision. The nuance between analyst and scientist is less clear. Can you describe what type of candidates the two draws or what you look for depending on the title?
  - bigger_cheese 10y ago
    My job title is currently "Data Engineer" I work in an industrial plant. Here's my two cents: My background is in Engineering (I'm a materials engineer by qualification). What differentiates me from a statistician, analyst etc is my domain knowledge. I have almost 15 years experience working with industrial processes. I have the background knowledge of chemistry, thermodynamics, mechanics etc. Which someone with a stats background would be lacking. So when I am asked to optimize an industrial process I can utilize that expertise whilst developing models. I would expect that a data scientist would know more about machine learning and would have a much stronger stats background than me. They'd also probably write much better code (I work in C/C++ and SAS, from what I have seen data scientists tend to be Python/R focused).
  - l_t 10y ago
    Not the OP, but in my experience a "Data analyst" is mostly responsible for writing analytical SQL queries and generating reports. So they don't require a strong math background or programming skills (other than SQL).