3 ms·

Yeah excel based data ingest is a pretty brutal problem to solve.

• hermitcrab 2y ago
  We can automatically handle some schema drift if columns are renamed or reordered, or columns added or deleted. But if they are both renamed AND re-ordered, you are out of luck!
  • bartread 2y ago
    If you detect a level of drift that you can't handle, this is the perfect opportunity to delegate that bit of work to an LLM, if it's a problem that you deal with regularly enough to feel the cost of it to your business. The latest generation of LLMs are pretty, actually very, good at this kind of situation, where somebody has renamed something - but kept some semblance of the meaning - and also moved it so a basic, or even a fuzzy, comparison might not be able to make a good match. But a model like GPT-4o-mini will eat a problem like this for breakfast, and it's now incredibly cheap to use it for this kind of thing as well.