5 ms·

Based on my ~8 years of experience managing ML teams in big companies, I’d say it’s closer to 80% of cases. “You can’t deploy correctly because of data scienti

by mlthoughts2018 6y ago

Based on my ~8 years of experience managing ML teams in big companies, I’d say it’s closer to 80% of cases. “You can’t deploy correctly because of data scientists” is a failure of SRE organizations to provide support, tooling and training. In fact, most data science and ML engineers are quite skilled in systems engineering, because you have to do so much work with GPU hardware issues, underlying scientific package management, efficient data transportation, etc. Editing some kubernetes config, hardening a high traffic web service or optimizing a query based on an index are trivial by comparison, they are just boilerplate timewasters that need to be a different team’s job to automate.

ska 6y ago
There are companies who do this well, sure. Are you seriously suggesting that this applies 80% of the companies out there trying to apply ML methods? If so, it really doesn't match my experience in this and related fields. Of course we'll both have sampling bias here, so I could be missing the big picture; it's not like I've studied it industry(ies) wide. Failure or non-existence of effective SRE organizations is just one of many common failure modes ime.
- mlthoughts2018 6y ago
  Not necessarily 80% of companies, rather 80% of devops <> machine learning workflows in a given company.
amznthrowaway5 6y ago
"In fact, most data science and ML engineers are quite skilled in systems engineering, because you have to do so much work with GPU hardware issues, underlying scientific package management, efficient data transportation, etc." This does not align with my experience working with many data/applied scientists at very large companies. A lot of them cannot even write basic code or use git commands. The engineers who are capable are often silo'd away from the scientists, and the organizations struggle to produce any real value.
- mlthoughts2018 6y ago
  I’m sorry you’ve had such a rare and incredibly uncommon, unrepresentative experience. It sounds very idiosyncratic to your workplace and likely to the hiring processes.
  - amznthrowaway5 6y ago
    Why do you suspect this experience is so rare and unrepresentative, as opposed to yours? It is a company wide problem in the cases I've seen, the scientist positions are not even expected to have junior software engineer level competence in things like programming.
winchester6788 6y ago
> In fact, most data science and ML engineers are quite skilled in systems engineering, because you have to do so much work with GPU hardware issues, underlying scientific package management, efficient data transportation, etc. Unless your data scientists are expected to build the machines they use, they won't be dealing with any hardware issues at all. Literally every data scientist at big companies use pre-configured vms/notebooks in cloud.
- mlthoughts2018 6y ago
  This is just deeply wrong. Most data scientists hate Jupyter notebooks and deeply recognize the flaws of the paradigm, poor modularity or testability, etc. As an ML engineer you spend a lot of your time dealing with Cuda installations, custom compiler flags and then build/compilations of things like Tensorflow, deep internals of Docker image builds to make these environments reproducible, image processing software with opencv and tons of cross platform & software packaging headaches, writing efficient queries and understanding data structure implications for spark, arrow, hdfs, presto, postgres, etc etc, and standing up things like tensorboard for telemetry of ML training systems, deploying mlflow or kubeflow in kubernetes, and so on. The myth of data scientists as notebook jockeys is just one more symptom of the denial of SRE orgs to admit ML engineers are great system engineers, to try to control them with parochial devops requirements coming from outside specializations.
  - Izkata 6y ago
    > The myth of data scientists as notebook jockeys is just one more symptom of the denial of SRE orgs to admit ML engineers are great system engineers It's most obvious with this statement, but overall you seem to think "ML engineer == data scientist", which just isn't the case.
    - mlthoughts2018 6y ago
      That sounds like a No True Scotsman fallacy to me. You’re trying to define “data scientist” as someone who only knows how to use notebooks, fails to put testing as a first class consideration, etc., but that’s a severe minority of people with the job title of Data Scientist. I manage teams of both ML engineers and Data Scientists and have designed hiring processes for both within large ecommerce companies for years.

ska 6y ago

There are companies who do this well, sure. Are you seriously suggesting that this applies 80% of the companies out there trying to apply ML methods? If so, it really doesn't match my experience in this and related fields. Of course we'll both have sampling bias here, so I could be missing the big picture; it's not like I've studied it industry(ies) wide. Failure or non-existence of effective SRE organizations is just one of many common failure modes ime.

mlthoughts2018 6y ago
Not necessarily 80% of companies, rather 80% of devops <> machine learning workflows in a given company.

amznthrowaway5 6y ago

"In fact, most data science and ML engineers are quite skilled in systems engineering, because you have to do so much work with GPU hardware issues, underlying scientific package management, efficient data transportation, etc." This does not align with my experience working with many data/applied scientists at very large companies. A lot of them cannot even write basic code or use git commands. The engineers who are capable are often silo'd away from the scientists, and the organizations struggle to produce any real value.

mlthoughts2018 6y ago
I’m sorry you’ve had such a rare and incredibly uncommon, unrepresentative experience. It sounds very idiosyncratic to your workplace and likely to the hiring processes.
- amznthrowaway5 6y ago
  Why do you suspect this experience is so rare and unrepresentative, as opposed to yours? It is a company wide problem in the cases I've seen, the scientist positions are not even expected to have junior software engineer level competence in things like programming.

winchester6788 6y ago

> In fact, most data science and ML engineers are quite skilled in systems engineering, because you have to do so much work with GPU hardware issues, underlying scientific package management, efficient data transportation, etc. Unless your data scientists are expected to build the machines they use, they won't be dealing with any hardware issues at all. Literally every data scientist at big companies use pre-configured vms/notebooks in cloud.

mlthoughts2018 6y ago
This is just deeply wrong. Most data scientists hate Jupyter notebooks and deeply recognize the flaws of the paradigm, poor modularity or testability, etc. As an ML engineer you spend a lot of your time dealing with Cuda installations, custom compiler flags and then build/compilations of things like Tensorflow, deep internals of Docker image builds to make these environments reproducible, image processing software with opencv and tons of cross platform & software packaging headaches, writing efficient queries and understanding data structure implications for spark, arrow, hdfs, presto, postgres, etc etc, and standing up things like tensorboard for telemetry of ML training systems, deploying mlflow or kubeflow in kubernetes, and so on. The myth of data scientists as notebook jockeys is just one more symptom of the denial of SRE orgs to admit ML engineers are great system engineers, to try to control them with parochial devops requirements coming from outside specializations.
- Izkata 6y ago
  > The myth of data scientists as notebook jockeys is just one more symptom of the denial of SRE orgs to admit ML engineers are great system engineers It's most obvious with this statement, but overall you seem to think "ML engineer == data scientist", which just isn't the case.
  - mlthoughts2018 6y ago
    That sounds like a No True Scotsman fallacy to me. You’re trying to define “data scientist” as someone who only knows how to use notebooks, fails to put testing as a first class consideration, etc., but that’s a severe minority of people with the job title of Data Scientist. I manage teams of both ML engineers and Data Scientists and have designed hiring processes for both within large ecommerce companies for years.