4 ms·
Datetimes can come in thousands of formats, I think rule based tools would have a hard time detecting unknown formats perfectly. AI could be a potentially bette
by ethegwo 3y ago
Datetimes can come in thousands of formats, I think rule based tools would have a hard time detecting unknown formats perfectly. AI could be a potentially better way to parse datetimes, maybe not LLM though, what are your thoughts on this?
- syndicatedjelly 3y agoI haven't experienced these complex datetime situations personally. Yeah it can be a pain to figure out the formatting sometimes, but once it's known then you just roll with that format. I try to stick with unambiguous formats that are easily internationalized, such as 10-Jan-2023 instead of 10-1-2023 (could be Jan 10th or Oct 1st) If you have a situation where there are a variety of datetime formats in a single field, then I can see where an LLM approach could be interesting. But even then, I think the priority is to migrate all of those values to a unified format. After that, add sanitation and validation on the input side and database to make sure all newly entered datetimes are the same format.
- ethegwo 3y agoYes, I agree with you, governance is always more efficient than post-processing. But if humans are always diligent in data governance, we do not need the idea about "data lake"...