The boring answer is that it's context dependent, but fundamentally "do data engineering the same way you do high-performance software engineering". Have tests

The boring answer is that it's context dependent, but fundamentally "do data engineering the same way you do high-performance software engineering". Have tests that run fast, where fast means "a few seconds when you start" and "refactor as you go so the tests keep taking a tolerable amount of time". I think Kent Beck suggested 10 minutes in Extreme Programming Explained. We're gradually forming our own, complex opinions in this. In the consulting context, this is essentially our product. A fascinating realization moving from software to marketing is that a sales pitch or marketing strategy can be built in a way that isn't entirely dissimilar to code, and that it has second-order effects. They aren't the same because... they aren't the same, but there's an artistry combined with principles to doing it "right". And then as a consultant there's additional complexity, as each team is different. Some are high-performing and need a bit of an external jolt. Others need the help the most, but are in politicized environments so they're almost inaccessible until a new executive comes in who can admit there are problems (or indeed, even see that there are problems). Joe Reis has some great stuff in Fundamentals of Data Engineering, which includes advice on early objectives when rolling out a new practice. Disclaimer: Joe has hosted me on his podcast, and we are in the mutual-marketing whirlpool together. But I've been recommending his book long before I met him.