4 ms·

That does sound useful, but at the same time I can't help thinking that it would be better if the validation was pushed further up the stack, because trying to

• entee 6y ago
  I'd have loved that at the time! But clients don't like it when we ask them to fix these things. Less flippantly, with large organizations, the people giving you the data aren't necessarily the people who entered the data or have the ability or skills to fix it. They may also be legally prohibited from altering the data, so correcting errors can get tricky. Obviously, for downstream analytics you need good data, so we have to fix it as well as possible. Usually, it involved a lot of back and forth with the client. We'd tell them things like, "you have a lot of dates that are set as January 1, 1900, we'll treat those as undefined, does that make sense?" That's why a tool like Great Expectations, especially as it adds data introspection features can be super helpful. I can write a quick spec, compile a report and send the client that report in record time. Even without major errors, the client has all sorts of tribal knowledge about how to use that database is used, and often you have to deduce those rules or do good data introspection to ask the client the right questions.