4 ms·
Lots of homegrown dbs look like this. They started life as a doc, then a spreadsheet, got sucked into Access, and now are living in SQL Server or Oracle dbs.
by tom_b 14y ago
Lots of homegrown dbs look like this. They started life as a doc, then a spreadsheet, got sucked into Access, and now are living in SQL Server or Oracle dbs. I have seen this a bunch in research environments where the tech is a "one-off" to support the science. I usually do a little data cleaning and maybe try some data warehouse-style restructuring so that you can report on the existing data.
You might have luck replacing your current table mess with views (with the same name as the tables) while you fix the structure to a more normalized schema. But you might be better starting from scratch and migrating to a new "version" of the app while moving the data along to the new version.
As to your self-organizing data idea, I have wondered if something kind of cool could come out of probabilistic graphical models, but haven't dug into the theory of that (and there is the free cousera course I think) enough to say.
I am going to guess that what is going to get you in trouble with self-organizing data is that contextual relationships in the data that are currently interpreted by your end-users (and are important/critical to the user) will get lost in any algorithmic attempt to clean the data.
You might also want to investigate the social sciences data scene - those folks do lots of statistical/data hacking on qualitative data that involve weirdness about coding "grey area text answers" into quantitatively tractable code numbers. It looks and sounds like hell to me . . .