4 ms·
My understanding is that LLMs are more useful for making sense of unstructured information than making sense of structured information. Based on your explanatio
by syndicatedjelly 3y ago
My understanding is that LLMs are more useful for making sense of unstructured information than making sense of structured information. Based on your explanation, I now understand why you chose to do this experiment.
In the conclusion, the next steps you described were:
- Automatically detect column types
- Parse datetime
This is pretty easy to do with existing tools for a structured data set. What value is there to having an LLM do these tasks? I'm curious about how you might use an LLM for some more interesting use cases.
- ethegwo 3y agoDatetimes can come in thousands of formats, I think rule based tools would have a hard time detecting unknown formats perfectly. AI could be a potentially better way to parse datetimes, maybe not LLM though, what are your thoughts on this?
- syndicatedjelly 3y agoI haven't experienced these complex datetime situations personally. Yeah it can be a pain to figure out the formatting sometimes, but once it's known then you just roll with that format. I try to stick with unambiguous formats that are easily internationalized, such as 10-Jan-2023 instead of 10-1-2023 (could be Jan 10th or Oct 1st) If you have a situation where there are a variety of datetime formats in a single field, then I can see where an LLM approach could be interesting. But even then, I think the priority is to migrate all of those values to a unified format. After that, add sanitation and validation on the input side and database to make sure all newly entered datetimes are the same format.
- ethegwo 3y agoYes, I agree with you, governance is always more efficient than post-processing. But if humans are always diligent in data governance, we do not need the idea about "data lake"...