3 ms·

And yet we still haven't completely replaced those old, inflexible systems, even as we are inventing new ones. So, were relational databases and spreadsheets r

by easp 17y ago

And yet we still haven't completely replaced those old, inflexible systems, even as we are inventing new ones. So, were relational databases and spreadsheets really the answer, or just an alternative?

fauigerzigerk 17y ago
Nothing is ever completely replaced, but you can answer your own question by asking _why_ the old systems were inflexible and _how_ RDBMS and spreadsheets improved that situation. It's not just some historical freak event like Betamax v VHS. There's logic. Hard coded access paths and data structures combined with explicit and selective consistency management is always faster and more scalable provided the number of different use cases is small and requirements don't change much. Doing a small number of things on a huge scale justifies the approach that people like twitter are taking. Unfortunately what scales physically doesn't scale in terms of logical complexity. As soon as a greater variety of tasks, views and analyses have to be supported, productivity plummets and things come to a screeching halt. That's exactly what happened to corporate IT departments back then. Everything took ages to implement because you couldn't just join over or group by some unexpected set of attributes. As a reaction to that, two things happened. For one, people tried to free their data from the iron grip of centralised IT departments and put it on PCs and into spread sheets that used very generic data structures to support a very large variety of views and analyses. Nothing had to be predetermined. Of course that also led to utter chaos in terms of consistency and it didn't scale to large amounts of data. The other idea to cope with hard coded information was to separate the logical representation of information from pysical representations and access paths. Relational normalisation doesn't just guarantee consistency, much more importantly it guarantees the productivity of creating a wide variety of views on data. A separate and independent layer contains all the special casing and optimisations for frequent use cases. I think the idea of modelling information according to a general, formally well defined logic and separating that model from implementation constraints is a timeless one. It's not just "an alternative". It's a fundamental principle of dealing with information and it's not specific to the relational model. I've been doing this for long enough to know that this kind of purity is hard to achieve and has to be comprimised now and then. Throwing out good design principles is sometimes necessary. I can see why it is necessary for Google, Amazon or Facebook. But it is going to make these behemoths slower and less flexible. It's a price they pay for their size. It makes no sense for small companies to pay that price whilst not benefiting from the same economies of scale.

by easp 17y ago

fauigerzigerk 17y ago

Nothing is ever completely replaced, but you can answer your own question by asking _why_ the old systems were inflexible and _how_ RDBMS and spreadsheets improved that situation. It's not just some historical freak event like Betamax v VHS. There's logic. Hard coded access paths and data structures combined with explicit and selective consistency management is always faster and more scalable provided the number of different use cases is small and requirements don't change much. Doing a small number of things on a huge scale justifies the approach that people like twitter are taking. Unfortunately what scales physically doesn't scale in terms of logical complexity. As soon as a greater variety of tasks, views and analyses have to be supported, productivity plummets and things come to a screeching halt. That's exactly what happened to corporate IT departments back then. Everything took ages to implement because you couldn't just join over or group by some unexpected set of attributes. As a reaction to that, two things happened. For one, people tried to free their data from the iron grip of centralised IT departments and put it on PCs and into spread sheets that used very generic data structures to support a very large variety of views and analyses. Nothing had to be predetermined. Of course that also led to utter chaos in terms of consistency and it didn't scale to large amounts of data. The other idea to cope with hard coded information was to separate the logical representation of information from pysical representations and access paths. Relational normalisation doesn't just guarantee consistency, much more importantly it guarantees the productivity of creating a wide variety of views on data. A separate and independent layer contains all the special casing and optimisations for frequent use cases. I think the idea of modelling information according to a general, formally well defined logic and separating that model from implementation constraints is a timeless one. It's not just "an alternative". It's a fundamental principle of dealing with information and it's not specific to the relational model. I've been doing this for long enough to know that this kind of purity is hard to achieve and has to be comprimised now and then. Throwing out good design principles is sometimes necessary. I can see why it is necessary for Google, Amazon or Facebook. But it is going to make these behemoths slower and less flexible. It's a price they pay for their size. It makes no sense for small companies to pay that price whilst not benefiting from the same economies of scale.