3 ms·

The problem with Data Scientists is that they have been historically overpaid while producing poor quality code. Mostly data janitors who create transformation

The problem with Data Scientists is that they have been historically overpaid while producing poor quality code. Mostly data janitors who create transformation code using open source libraries, converting from one format / database / stream to another format / database / stream. Then real software engineers look at their code and it’s janky and messed up. I no longer hire data scientists. I hire recent CS graduates then they cut their teeth on that work because it is very easy, typically low risk, and they can learn the basics of software engineering.

CalRobert 3y ago
"Here's a pile of crap in a jupyter notebook, turn it in to a product!"
- datavirtue 3y ago
  Same as: "Here is a pile of crap in a spreadsheet, turn it into a product!" That is our job.
- rectang 3y ago
  That describes a past gig of mine which was fantastically successful. I was handed a messy 3000-line Jupyter notebook which contained the prototype of a great product, and given the task of productionizing it. In fact, this is what I've done at several startups now: take prototypes and productionize them. At this point, I'm almost a specialist in transitioning from early stage to growth stage. I can do prototyping, but I recognize that productionizing is my stronger suit. The technology for the prototype doesn't matter — Jupyter is fine. In fact, it's better if the prototype is shite because then it makes it easier to make it the "one to throw away".
- datadrivenangel 3y ago
  I once watched a data scientist copy code from a notebook into an email and send it to us, right before going to Hawaii for two weeks so we could productize it. This was after advocating for version control for the project for months...
claytonjy 3y ago
Yup, between analytics engineers and ML engineers, it seems data scientists are the weird middle we no longer need. I have worked with "data scientists" that are more rebranded statisticians. Some are overpaid SASS users, but some are true statistical wizards who can also program (R/Python/SQL), and when you need them they're great. I don't need them to write production code. I'd rather call them statisticians but I'm glad to see them getting paid better!

CalRobert 3y ago

"Here's a pile of crap in a jupyter notebook, turn it in to a product!"

datavirtue 3y ago
Same as: "Here is a pile of crap in a spreadsheet, turn it into a product!" That is our job.
rectang 3y ago
That describes a past gig of mine which was fantastically successful. I was handed a messy 3000-line Jupyter notebook which contained the prototype of a great product, and given the task of productionizing it. In fact, this is what I've done at several startups now: take prototypes and productionize them. At this point, I'm almost a specialist in transitioning from early stage to growth stage. I can do prototyping, but I recognize that productionizing is my stronger suit. The technology for the prototype doesn't matter — Jupyter is fine. In fact, it's better if the prototype is shite because then it makes it easier to make it the "one to throw away".
datadrivenangel 3y ago
I once watched a data scientist copy code from a notebook into an email and send it to us, right before going to Hawaii for two weeks so we could productize it. This was after advocating for version control for the project for months...

claytonjy 3y ago

Yup, between analytics engineers and ML engineers, it seems data scientists are the weird middle we no longer need. I have worked with "data scientists" that are more rebranded statisticians. Some are overpaid SASS users, but some are true statistical wizards who can also program (R/Python/SQL), and when you need them they're great. I don't need them to write production code. I'd rather call them statisticians but I'm glad to see them getting paid better!