The most important DE skill you should learn is how to fix data problems when they happen. Data problems include: duplicate rows/events, upstream data errors th

The most important DE skill you should learn is how to fix data problems when they happen. Data problems include: duplicate rows/events, upstream data errors that need to be corrected, error in your ETL logic, etc. You need to build tools to reason about the graph of data dependencies, i.e. downstream/upstream dependencies of datasets, and create a plan for repair. Datasets need to repaired in layers, starting from the ones closest to the source and working downstream to the leaves. Some datasets are self-healing or not (i.e. they're rebuilt from scratch when their inputs change), so you just let those rebuild. Incremental datasets are the worst to repair because they don't restate history and you have to fork them from the point when a problem happened and rebuild from that point onwards. Without tools to help you repair data, sooner or later you'll run into a problem that will take you days (even weeks) to fix, and your stakeholders will be breathing down your neck.