31 ms·

I'm curious what innovative tools have emerged in the past two years that changed the dynamic?

• haney 8y ago
  Not sure what the author was referring to but, Airflow has gotten better / gained wider adoption during that time and my team started using DBT which saved a bunch time and was new during that period.
  • closeparen 8y ago
    Airflow makes the distinction wider, as creating data pipelines requires even less software engineering.
• iblaine 8y ago
  Airflow and tools like it are probably the biggest reason for the shift. Another issue is the need to integrate different technologies requires having the skills of a software engineer. When the landscape for tech in DE was oracle, mysql and Cognos, DE's didn't need to know about OOP or consensus algorithms. Because the landscape now includes hadoop, redshift, kafka, spark, airflow, notebooks, TiDB and lord knows what else, DE's need to have most of the skills of a software engineer to be successful.