3 ms·

I don't disagree with anything you said, but I think you are underindexing on the importance of being a resilient data organization. Few things reduce customer

I don't disagree with anything you said, but I think you are underindexing on the importance of being a resilient data organization. Few things reduce customer trust like having the same error happen multiple times, especially when consumers are the ones identifying the problem. Additionally, changes in the source data are quite common and very hard to detect. I agree that changes in code are another source of the issue, but that is usually under control of the data team, which is a totally different type of error compared to things outside your locus. Automated statistical tests don't work because variance is common even in stable data streams; these add too much noise. It's critical for an organization to be able to patch a hole after it's been found, and this is where I think SQL tests shine. They need to be really easy to implement and monitor, but they serve a critical purpose for ensuring continued operational quality and downstream trust.

tqi 3y ago
Hm I think you and I might actually be saying the same thing, just using different terms? Tests as I understand them (based on reading the article shared by the person I was responding to[1]) run on mock data, rather than real source data. Those are the ones that I feel have limited ROI. I definitely agree there is a lot of value in tests that check against actual source data (which that person's link refers to as audits[1]). [1] https://sqlmesh.readthedocs.io/en/stable/concepts/tests/ https://sqlmesh.readthedocs.io/en/stable/concepts/tests/ [2] https://sqlmesh.readthedocs.io/en/stable/concepts/audits/ https://sqlmesh.readthedocs.io/en/stable/concepts/audits/
- edmundsauto 3y ago
  Ah yes, I think we are in agreement :). Understanding terms of art becomes harder every year - the differentiation between tests and audits makes sense but adds so much friction as a noob that I wish they had just stuck to prior art.

tqi 3y ago

Hm I think you and I might actually be saying the same thing, just using different terms? Tests as I understand them (based on reading the article shared by the person I was responding to[1]) run on mock data, rather than real source data. Those are the ones that I feel have limited ROI. I definitely agree there is a lot of value in tests that check against actual source data (which that person's link refers to as audits[1]). [1] https://sqlmesh.readthedocs.io/en/stable/concepts/tests/ https://sqlmesh.readthedocs.io/en/stable/concepts/tests/ [2] https://sqlmesh.readthedocs.io/en/stable/concepts/audits/ https://sqlmesh.readthedocs.io/en/stable/concepts/audits/

edmundsauto 3y ago
Ah yes, I think we are in agreement :). Understanding terms of art becomes harder every year - the differentiation between tests and audits makes sense but adds so much friction as a noob that I wish they had just stuck to prior art.