It is the bare minimum to do data tests throughout your pipeline, but a good, open source, structured framework that I don’t have to write myself is kinda nice

It is the bare minimum to do data tests throughout your pipeline, but a good, open source, structured framework that I don’t have to write myself is kinda nice ;) I can then separate testing from the transformation code (which yes, should be tested too in unit tests and the like), and use this tool as an audit to say, “did my code do the right thing on the dataset in this latest ETL?”, and “where did it screw up?” You can also use it to write a specific data spec and rely on the tool to guarantee that spec. It’s a lot easier to write a spec than the tool that does the checking. That spec also documents your data standard, is machine readable, and versionable. Ideally the cost is low, you spend 15m with the client data dictionary, write the spec, and then go have a coffee while the tool sees and highlights where the dataset is out of spec.