4 ms·

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

watermanio 6y ago
This feels especially true when you have access to things like BigQuery ML. It's very easy for an average engineer (like me) to start using ML using these tools, but a lot harder to explain how it works, or exactly which type of models to use. In my mind a DS would be really useful to just point us in the right direction and check work. Like a super specialist QA...
listenallyall 6y ago
> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own from scratch -- it's not going to happen, they just don't have that skill, the instructions on the box don't account for that possibility. As soon as the task diverges from the simple step 1, step 2, step 3 that they were originally taught, you realize they have very little ability to adapt. YMMV of course.
- tchalla 6y ago
  > Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe work, don't hire people will recipe skills. Do not have job interviews that select for recipe people. But, that is exactly what most companies do.
  - simo7 6y ago
    Precisely.
- superhuzza 6y ago
  >Asking them to clean and scrub the source data set...it's not going to happen, they just don't have that skill I think you've been working with conmen/conwomen. I've never seen a data science project that doesn't involve data cleaning or wrangling of some sort.
  - listenallyall 6y ago
    Have you read through the comment thread? Did you read the article? Most everyone is in agreement that projects require a lot of cleaning & wrangling and a lot more -- the point is that data scientists are generally not doing that stuff, they expect academic-quality, pre-processed, pristine data, so it's data engineers who are stuck preparing the data, and who are in high demand.
    - superhuzza 6y ago
      Yes I read both the article and the comments. I meant a data science project in terms of a project completed by data scientists. In my experience, all data scientists are accustomed to doing extensive cleaning etc.
- oivey 6y ago
  Yep. The key is really software skills. If you’re unable to even filter the data yourself, you’re also probably unlikely to be able do implement novel analysis techniques, especially if the analysis algorithm has many complicated steps or is computationally expensive. In all fairness, it’s basically impossible for a new grad to have those skills. 4 years of a bachelors in any field isn’t enough to cover such a wide area. Even for people with graduate degrees it’s a stretch.
  - jimbokun 6y ago
    The hope is that the 4 year degree gave you the ability to quickly pick up those skills on your own. If your four year degree didn't give you the ability to learn and expand your knowledge on your own, its a colossal waste of your time and money.
    - oivey 6y ago
      Sure, but depending on what you’re doing, “quick” might be years. You can get a PhD in understanding the theory, a PhD in designing fast numerical algorithms, or spend many years becoming a strong software engineer. I think the willingness to learn a diverse set of things is much more important than learning narrow areas fast. The short length of a bachelors usually isn’t enough to get this diversity.

watermanio 6y ago

This feels especially true when you have access to things like BigQuery ML. It's very easy for an average engineer (like me) to start using ML using these tools, but a lot harder to explain how it works, or exactly which type of models to use. In my mind a DS would be really useful to just point us in the right direction and check work. Like a super specialist QA...

listenallyall 6y ago

> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own from scratch -- it's not going to happen, they just don't have that skill, the instructions on the box don't account for that possibility. As soon as the task diverges from the simple step 1, step 2, step 3 that they were originally taught, you realize they have very little ability to adapt. YMMV of course.

tchalla 6y ago
> Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe work, don't hire people will recipe skills. Do not have job interviews that select for recipe people. But, that is exactly what most companies do.
- simo7 6y ago
  Precisely.
superhuzza 6y ago
>Asking them to clean and scrub the source data set...it's not going to happen, they just don't have that skill I think you've been working with conmen/conwomen. I've never seen a data science project that doesn't involve data cleaning or wrangling of some sort.
- listenallyall 6y ago
  Have you read through the comment thread? Did you read the article? Most everyone is in agreement that projects require a lot of cleaning & wrangling and a lot more -- the point is that data scientists are generally not doing that stuff, they expect academic-quality, pre-processed, pristine data, so it's data engineers who are stuck preparing the data, and who are in high demand.
  - superhuzza 6y ago
    Yes I read both the article and the comments. I meant a data science project in terms of a project completed by data scientists. In my experience, all data scientists are accustomed to doing extensive cleaning etc.
oivey 6y ago
Yep. The key is really software skills. If you’re unable to even filter the data yourself, you’re also probably unlikely to be able do implement novel analysis techniques, especially if the analysis algorithm has many complicated steps or is computationally expensive. In all fairness, it’s basically impossible for a new grad to have those skills. 4 years of a bachelors in any field isn’t enough to cover such a wide area. Even for people with graduate degrees it’s a stretch.
- jimbokun 6y ago
  The hope is that the 4 year degree gave you the ability to quickly pick up those skills on your own. If your four year degree didn't give you the ability to learn and expand your knowledge on your own, its a colossal waste of your time and money.
  - oivey 6y ago
    Sure, but depending on what you’re doing, “quick” might be years. You can get a PhD in understanding the theory, a PhD in designing fast numerical algorithms, or spend many years becoming a strong software engineer. I think the willingness to learn a diverse set of things is much more important than learning narrow areas fast. The short length of a bachelors usually isn’t enough to get this diversity.