4 ms·

> Normalisation matters to transaction processing. If you mean to mean that normalization matters only for transactional processing, I dare disagree. Normaliza

by gbog 15y ago

> Normalisation matters to transaction processing. If you mean to mean that normalization matters only for transactional processing, I dare disagree. Normalization matters for data sanity. Building and maintaining a complex and agile application is much easier on properly normalized data. Making sure one bit of data is stored only in one place and is properly decoupled from other bits of data's existence is still the Right Thing to do. True, it is sometime in direct contradiction with data access performance, but this is an optimization issue, which can be solved with denormalization, materialization or the use of some "NoSQL" storage. Anyway, in these NoSQL discussions I always wonder: As far as I know, Wikipedia is still using a purely relational data-store, and it's main data (an enormous dict of blobs) seem to be the perfect candidate for a non-relational storage. So then, if NoSQL is so good at this task, how come they don't have moved yet? Should they move? (Genuine question)

neilk 15y ago
That's a complicated topic. For the social web, people typically denormalize data because consistency is less important than personalization and/or read performance. At some point, it's just easier to write the same data in a redundant way than to query from a canonical source and transform it on the fly. Wikipedia's raison d'etre is to show the same data to everybody. In fact, it goes to great lengths to ensure that everybody sees the very last updated version, no matter what. So there's no great pressure to denormalize core services; in fact, quite the opposite. The next part of your question is whether a document-oriented database would be better. I think it would be possible to write a wiki on top of a document store. You'd gain a lot from simplicity, although you'd lose certain kinds of flexibility. But, this is not practical for Wikipedia at this point. For everything else that goes into rebuilding a page, or administration, there are plenty of traditional joins. The software is very much married to SQL. MediaWiki, the software that powers Wikipedia, is open source and database agnostic. There are people running MediaWiki sites on pretty much every RDBMS you can name. While queries and updates are all abstracted away, the core concepts are all obviously SQL. The abstraction layer just gets rid of syntactical quirks and handles escaping. That said, a typical MediaWiki installation is not well normalized either. MediaWiki is capable of hosting a lot of extensions that extend the behavior of the wiki. If the extension needs to persist data that's associated with existing tables, a typical strategy is for the plugin to maintain its own parallel tables of data, which reuse the same primary key. Like a lot of successful websites, the MediaWiki culture is pragmatic above all else. SQL databases are used to persist data and the best you could say is that it's a hybrid strategy.
jacques_chester 15y ago
> If you mean to mean that normalization matters only for transactional processing, I dare disagree. I think we are, actually, in agreement. The benefits of ACID are essential to OLTP. It also so happens that normalised data can be written fasted as it only needs to be written in once per datum. As for Wikipedia, I think there's two reasons. First would be path dependence, second would be that their data is (I'm guessing) easily partitionable.

by gbog 15y ago

neilk 15y ago

That's a complicated topic. For the social web, people typically denormalize data because consistency is less important than personalization and/or read performance. At some point, it's just easier to write the same data in a redundant way than to query from a canonical source and transform it on the fly. Wikipedia's raison d'etre is to show the same data to everybody. In fact, it goes to great lengths to ensure that everybody sees the very last updated version, no matter what. So there's no great pressure to denormalize core services; in fact, quite the opposite. The next part of your question is whether a document-oriented database would be better. I think it would be possible to write a wiki on top of a document store. You'd gain a lot from simplicity, although you'd lose certain kinds of flexibility. But, this is not practical for Wikipedia at this point. For everything else that goes into rebuilding a page, or administration, there are plenty of traditional joins. The software is very much married to SQL. MediaWiki, the software that powers Wikipedia, is open source and database agnostic. There are people running MediaWiki sites on pretty much every RDBMS you can name. While queries and updates are all abstracted away, the core concepts are all obviously SQL. The abstraction layer just gets rid of syntactical quirks and handles escaping. That said, a typical MediaWiki installation is not well normalized either. MediaWiki is capable of hosting a lot of extensions that extend the behavior of the wiki. If the extension needs to persist data that's associated with existing tables, a typical strategy is for the plugin to maintain its own parallel tables of data, which reuse the same primary key. Like a lot of successful websites, the MediaWiki culture is pragmatic above all else. SQL databases are used to persist data and the best you could say is that it's a hybrid strategy.

jacques_chester 15y ago

> If you mean to mean that normalization matters only for transactional processing, I dare disagree. I think we are, actually, in agreement. The benefits of ACID are essential to OLTP. It also so happens that normalised data can be written fasted as it only needs to be written in once per datum. As for Wikipedia, I think there's two reasons. First would be path dependence, second would be that their data is (I'm guessing) easily partitionable.