3 ms·

So, basically a test step in your ingest pipeline? I was under the impression that it is the bare minimum if we are talking about production. Why is a separate

So, basically a test step in your ingest pipeline? I was under the impression that it is the bare minimum if we are talking about production. Why is a separate tool necessary? To normalize the pipeline tests? If that's the case maybe there's another issue with engineering principles and basics that is not adressed but I don't know. I don't want to just be sceptical and dismiss anyone's effort for a free and open source tool but since I can express what I think I believe it will yield more overhead than anything, at least for my use cases so far since testing the data require way more steps than analyzing a single dump for coherence. While I can see the advantage of such a tool with exchanging interface definitions with data providers or other people, I consider it yet another thing to add to the whole infrastructure which obviously comes at a cost that is not 0 that can ultimately be solved with documentation enforced through sane devops methods.

entee 6y ago
It is the bare minimum to do data tests throughout your pipeline, but a good, open source, structured framework that I don’t have to write myself is kinda nice ;) I can then separate testing from the transformation code (which yes, should be tested too in unit tests and the like), and use this tool as an audit to say, “did my code do the right thing on the dataset in this latest ETL?”, and “where did it screw up?” You can also use it to write a specific data spec and rely on the tool to guarantee that spec. It’s a lot easier to write a spec than the tool that does the checking. That spec also documents your data standard, is machine readable, and versionable. Ideally the cost is low, you spend 15m with the client data dictionary, write the spec, and then go have a coffee while the tool sees and highlights where the dataset is out of spec.

entee 6y ago

It is the bare minimum to do data tests throughout your pipeline, but a good, open source, structured framework that I don’t have to write myself is kinda nice ;) I can then separate testing from the transformation code (which yes, should be tested too in unit tests and the like), and use this tool as an audit to say, “did my code do the right thing on the dataset in this latest ETL?”, and “where did it screw up?” You can also use it to write a specific data spec and rely on the tool to guarantee that spec. It’s a lot easier to write a spec than the tool that does the checking. That spec also documents your data standard, is machine readable, and versionable. Ideally the cost is low, you spend 15m with the client data dictionary, write the spec, and then go have a coffee while the tool sees and highlights where the dataset is out of spec.