The "data engineer" became a distinct role to bring over Software Engineering practices to data processing; such as those practices are, they were a marked impr

The "data engineer" became a distinct role to bring over Software Engineering practices to data processing; such as those practices are, they were a marked improvement over their absence. Building a bridge from one shore to the other with application programming languages and data processing tools that worked much closer to other forms of programming was a huge part of that push. Of course, the big data tools were intricate machines that were easy to learn and very hard to master, and data engineers had to be pretty sophisticated. So, it became cheaper to move much of that apparatus to data warehouses and, as you said, commoditize the building of data pipelines that way. Software is as widespread as it is today because in every generation the highly skilled priestly classes that were needed to get the job done were displaced by people with less training enabled by new tools or hardware; else it'd be all rocket simulations done by PhD physicists still. But the technical debt will be hefty from this shift.