3 ms·
What's "this kind of work"? :) The company I work for does reconciliation as a service, something that's pervasive in finance. We support customer-defined schem
by barrkel 9y ago
What's "this kind of work"? :) The company I work for does reconciliation as a service, something that's pervasive in finance. We support customer-defined schemas; in fact that's one of our selling-points. I could start talking about the kinds of things we've done with MySQL to make this perform well - it's not difficult, just a bit unorthodox.
Anyway, the diversity in customer schema leaks out into the Hadoop schema, where we'd much prefer to give customers data using column names they're familiar with, and we also want to give them rows from all their different schemas in a single table (because many schemas have overlap by design). The superset of all schema columns is large, however. The problem can be overcome with more tooling - defining friendly views with explicit column choice - but having the option to implement that (and go to market sooner), vs a requirement to implement that, adds up to a distinct advantage for tech that can support the extra columns.
- jamespo 9y agoEdgar F. Codd just started spinning
- barrkel 9y agoSomething you need to bear in mind is that distributed joins are very expensive; you have a better time designing your schema such that related data can be placed logically close together, whether it's arrays / maps inside rows (for one to many), or very wide rows (denormalizing what might be a star schema). (I know, in a column store having related data in another column isn't actually close together; but it can be stepped through at the same time, it doesn't need a join to be correlated, it's correlated naturally.)
- YCode 9y agoTo be honest, "this kind of work" was a euphemism for "Just what the hell are you doing that requires over a thousand columns?" I understand though that sometimes customers give you a rotting dumpster of data and ask for critical insight into their operations.