4 ms·

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Dat

by sgt101 6y ago

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.

legerdemain 6y ago
Sure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers. Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it in a box and sell it at commodity software prices?
- oivey 6y ago
  If something like that becomes mainstream, data scientists who just glue together canned ML algorithms from libraries should also be concerned. The kind of automation that enables eliding finding and cleaning the right data probably can also manage running PCA and basic ML.
  - sgt101 6y ago
    I remember the same claims for ISL Clementine before IBM bought it and turned it into SPSS modeler.
lumost 6y ago
The way the evolution in software went, platforms became more capable and allowed individuals to automate more common tasks. QA/DevOps/SRE teams were consolidated and replaced with smaller platform teams which empowered internal engineers to quickly write scalable and well tested services. If data management, instrumentation, and ML tooling become sufficient then perhaps the data engineers will be replaced by a science platform team. Caveat is that many scientists are expected to publish novel research to advance their career. "Infrastructure" and "data management" do not tend to produce the kinds of sexy projects which are attractive to publish.
- sgt101 6y ago
  I think this model (an integrated team) is what I see, there is a huge benefit in terms of short decision loops from having one team - but data engineering skills are really important in enabling it. Also if the people doing the data engineering are close to the data science then there's much more likelihood that they will produce effective solutions for the backend of the project.
  - lumost 6y ago
    aye - this pattern often trends towards the roles merging over time. The counter example to the platform team approach from the software side is Software Engineers owning the full infra and ops that their services need. There's a big incentive for companies that need to hire more people to give the "better" title out for the same kind of work. If managers do maintain a gold/silver role on their team all of the folks in the silver role will look at the gold role as their next move. Worse there is a net-negative productivity drag where the gold/silver role constantly debate what's in-scope vs. out of scope for their work. I once saw a team where the scientists were meant to be equivalent to SDEs in coding skill, but the scientists could only in practice do some light python/bash scripting. They tried to make the SDEs responsible for "productionalizing" the projects which meant adding tests/etc. The engineers who could all left the team in 6 months, the ones who remained were also unable to perform more than light bash scripting/python work.

by sgt101 6y ago

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.

legerdemain 6y ago

Sure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers. Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it in a box and sell it at commodity software prices?

oivey 6y ago
If something like that becomes mainstream, data scientists who just glue together canned ML algorithms from libraries should also be concerned. The kind of automation that enables eliding finding and cleaning the right data probably can also manage running PCA and basic ML.
- sgt101 6y ago
  I remember the same claims for ISL Clementine before IBM bought it and turned it into SPSS modeler.

lumost 6y ago

The way the evolution in software went, platforms became more capable and allowed individuals to automate more common tasks. QA/DevOps/SRE teams were consolidated and replaced with smaller platform teams which empowered internal engineers to quickly write scalable and well tested services. If data management, instrumentation, and ML tooling become sufficient then perhaps the data engineers will be replaced by a science platform team. Caveat is that many scientists are expected to publish novel research to advance their career. "Infrastructure" and "data management" do not tend to produce the kinds of sexy projects which are attractive to publish.

sgt101 6y ago
I think this model (an integrated team) is what I see, there is a huge benefit in terms of short decision loops from having one team - but data engineering skills are really important in enabling it. Also if the people doing the data engineering are close to the data science then there's much more likelihood that they will produce effective solutions for the backend of the project.
- lumost 6y ago
  aye - this pattern often trends towards the roles merging over time. The counter example to the platform team approach from the software side is Software Engineers owning the full infra and ops that their services need. There's a big incentive for companies that need to hire more people to give the "better" title out for the same kind of work. If managers do maintain a gold/silver role on their team all of the folks in the silver role will look at the gold role as their next move. Worse there is a net-negative productivity drag where the gold/silver role constantly debate what's in-scope vs. out of scope for their work. I once saw a team where the scientists were meant to be equivalent to SDEs in coding skill, but the scientists could only in practice do some light python/bash scripting. They tried to make the SDEs responsible for "productionalizing" the projects which meant adding tests/etc. The engineers who could all left the team in 6 months, the ones who remained were also unable to perform more than light bash scripting/python work.