3 ms·
Instead of sneering at "having to deal with" data scientists, consider that the data scientists themselves would often much rather have data engineers and dev o
by nerdponx 6y ago
Instead of sneering at "having to deal with" data scientists, consider that the data scientists themselves would often much rather have data engineers and dev ops people involved in the process.
Data scientists like to quip that 80% of the job is data cleaning, with the remaining 20% divided up arbitrarily among other tasks as suited the joke. In some shops nowadays, it's more like 45% data cleaning, 45% data engineering/ops/programming just trying to make your results available to the rest of your organization, and 10% research.
If I can spend less time learning/doing software engineering and devops and more time doing actual data science, that's great. At a previous job, my team was clamoring for more data engineer hiring, and part of the reason our projects were slipping and starting to fail was lack of data engineering support. Our tooling was shit, our processes were shit, our code was shit, and access to (and trust of) our data sources was especially wet and stinky shit.
It made the daily work of doing data science a miserable slog of ad-hoc duct-tape solutions, and it contributed to us being generally ineffective as a team.
All of this would have been fixed if we had one competent data engineer with some actual real-world data/ML engineering experience and good communication/advocacy skills. Let alone two or three!
- ramraj07 6y agoIf the DE tooling was shit and you couldn't hire more fast enough, why didn't your team members start addressing these problems? Surely spending half the time cleaning up the pipes would increase the value of what you do with the other half?