3 ms·
Listening to Software Engineering Daily, I heard that the there is a trend toward developing 'data engineering' as a discipline that handles sanitizing the data
by brudgers 10y ago
Listening to Software Engineering Daily, I heard that the there is a trend toward developing 'data engineering' as a discipline that handles sanitizing the data pipeline so that data scientists can spend more time working at business abstraction layers.
There's probably a scale at which that works better, two pizza data science teams for example.
- daemonk 10y agoMy issue with these sanitizing pipelines is that they only really work for fields where the data generation is relatively stable. Meaning, whatever instrument/method used in generating the data doesn't change dramatically every year. It's extremely challenging to design a all-purpose sanitizing pipeline. So I can imagine a pipeline developed for use with well established social media APIs or standard scientific experiments that have been in used for decades. But it is hard to imagine a pipeline that can handle amorphous emerging high throughput instruments/methods.
- randcraw 10y agoYeah, DE seems to have emerged as the title for those who manage large databases -- who regularly import raw data, cleanse, normalize, update, construct analysis pipelines, scale up and parallelize processes, and validate results. It lies somewhere between a DBA, sysadmin, and HPC engineer, with an awareness of basic stats -- where mastery of Hadoop's many components might converge. It seems like a natural evolution of DB admin for very large scale noSQL and DBs, esp. those with unstructured data and often non-commodity architecture. I've seen numerous companies looking for such folks and I suspect demand will rise. IMO, it's not a job you'd want to outsource. The role is too mission critical, the skills not predictable enough to be a commodity, and the penalty for screwing up is too great.