3 ms·

Let me offer the more blunt materialist analysis: Data engineers are being deskilled into data analysts and too blinded by shiny cloud advertisements to notice.

• forgetfulness 4y ago
  The "data engineer" became a distinct role to bring over Software Engineering practices to data processing; such as those practices are, they were a marked improvement over their absence. Building a bridge from one shore to the other with application programming languages and data processing tools that worked much closer to other forms of programming was a huge part of that push. Of course, the big data tools were intricate machines that were easy to learn and very hard to master, and data engineers had to be pretty sophisticated. So, it became cheaper to move much of that apparatus to data warehouses and, as you said, commoditize the building of data pipelines that way. Software is as widespread as it is today because in every generation the highly skilled priestly classes that were needed to get the job done were displaced by people with less training enabled by new tools or hardware; else it'd be all rocket simulations done by PhD physicists still. But the technical debt will be hefty from this shift.
• acjohnson55 4y ago
  At the end of the day, it's about providing value to businesses. If the same value can be provided with less intensive skillsets and more efficiently, this is a good thing.