6 ms·

The NoSQL data model - please read as document oriented - is fundamentally flawed because it forces you into denormalization without good reasons like performan

by danbruc 13y ago

The NoSQL data model - please read as document oriented - is fundamentally flawed because it forces you into denormalization without good reasons like performance optimization but with all the disadvantages. The only good reason not to use a relational database is that they suck at modeling complex entities and their relationships. Shattering nice entities into small relations to get them normalized, adding join tables to model many to many relationships and all that fun. Everything that is not roughly tree-like is just painful. More than once I finally decided to just stuff the really complex pieces into a blob - XML or what ever worked best - and just deal with it at the application layer. Not pretty, not fun, but still less painful than modeling and dealing with it in the relational model.

dhimes 13y ago
I'm with you here. I've found that even phones have enough juice to handle most manipulation, and the website appears more responsive because the server isn't bogged down. Haven't scaled it yet, though...
delluminatus 13y ago
I wouldn't say the model is fundamentally flawed. Looking at the roots of NoSQL (which I believe was first popularized by Google's BigTable), the original intent of a NoSQL database is to allow redundant storage of a lot of data on a DFS. They're meant for situations where you have data that doesn't fit well into a relational database model, and you need to run BIG queries (think PageRank) over the entire data in a fault-tolerant way. For instance thinking about PageRank, how many discrete data elements are in a single web page? This is "higher-order" data that in a relational database might have hundreds or thousands or tens of thousands of columns per row. Of course there are also in-memory transactional databases called NoSQL databases which are meant to be replacements for the traditional relational data store, but those are a very different beast from the original NoSQL data model.
- danbruc 13y ago
  That was bad and sensationalistic wording from my side. I should have said more clearly that I meant this in the context of modeling common business domains, not for specialized tasks that don't fit a relation model efficiently.
jchrisa 13y ago
One thing that is not easy to do with a relational model, is offline synchronization for high-availability on mobile devices. The document model was built for this, I find it hilarious that so few of the NoSQL databases are taking advantage of the latent capability. Try out Couchbase Mobile http://mobile.couchbase.com http://mobile.couchbase.com and compare it to best of breed relational sync.
- SchizoDuckie 13y ago
  It is not hard at all. people just hadn't built as much of the tools for it. Which also confuses me because we now have package managers that run on package managers. My own little WebSQL Object/Relation mapping library uses promises to get away from callback hell, and it's a pure joy to work with. It handles creating tables and inserting fixtures before the first connect promise is returned, so that you don't have to worry about the setup process and your data being available. Creating and inserting entities is as easy as > var P = new Presentation(); > P.set('name', 'test'); > P.Persist().then(function(result) { console.log('Done!') }); http://schizoduckie.github.io/CreateReadUpdateDelete.js/demo/ http://schizoduckie.github.io/CreateReadUpdateDelete.js/demo... I'm using it extensively in my Chrome plugin, and it runs on 10.000+ clients right now. Real world offline synchronisation example right here: https://github.com/SchizoDuckie/DuckieTV/blob/angular/js/services/MigrationService.js#L52 https://github.com/SchizoDuckie/DuckieTV/blob/angular/js/ser...
  - frik 13y ago
    Nice work. This would be the way to go. Sadly one cannot use it everywhere as WebSQL is not implemented in Firefox nor IE :( The good thing is WebSQL works also great on mobile devices.
- mamcx 13y ago
  I have think about use it, but still it fail if the data is relational. I have a invoice app, and none of the NOSQL I have evaluated guarantee the order of the updates or have atomic updates (need both). Is not fun if a invoice update partially or the related tables get out of sync. In the other hand, I'm stuck in how sync with a relational model :( So, it sync well and broke my relations and transactions or it sync hard but keep my data correct.
jt2190 13y ago
> The NoSQL data model - please read as document oriented > is fundamentally flawed because it forces you into > denormalization... Can you provide an example? How does it "force" this?
- danbruc 13y ago
  If you want to use a purely document oriented model and not have redundancies then you have to partition your object graph into a set of documents that do not overlap and have no relationships between them. I can not think of any real example that fits this model. Start anywhere, for example, pick your users and create a document for each. Now you take the orders of each user and also put them into the document belonging to the respective user. Fine, that works. Now we come to the products they ordered, and now we have a problem. Unless no product has ever been ordered by two different users we have to duplicate product information and put it into several different documents. And there is the same problem within documents if the user repeatedly ordered the same product. All interesting problem domains just don't fit the idea of piece of JSON without references, internally or externally. There is of course support for reference in document oriented databases, but then lets just call your documents rows in a table and your references foreign keys and we are almost back at the relational model besides that documents still have a richer model, a bit like a NF² relational model or database with deep support for XML or JSON columns.
  - jt2190 13y ago
    > Now you take the orders of each user and also put them > into the document belonging to the respective user. So the database doesn't force anything. The data modeler makes a questionable decision. > All interesting problem domains just don't fit the idea > of piece of JSON without references, internally or > externally. These databases won't prevent data modeling using references, they just often don't provide as much support as a relational database does for doing joins or enforcing key constraints.
    - danbruc 13y ago
      But that's my point - either you use documents in the way they are advertised and get redundancy or you normalize into many documents with relationships between them and then your documents just degenerate into entities. I am not saying there are no usecases where a document oriented model fits well, but I argue that it is not a good fit for modeling your average business.
tetha 13y ago
> The NoSQL data model - please read as document oriented These few words are pointing to a fundamental problem I'm already getting annoyed about, as little as I know about "NoSQL" databases so far: The term NoSQL-database is flawed, just as Fowler says in his introduction talk. Theres's document-oriented databases, and array-ish data storages (column family-oriented), there's graph based databases - and they have very, very different strengths. The article falls flat on that problem as well. Some data is not relational and you are correct that there is a strong overlap between document based storage systems and relational storage systems. However, I'm dealing with time series data, or with lock-graphs. I could shoe-horn those problems into a relational database, but after tinkering around with better storage engines for my problem... I'd call myself stupid for doing so. So, as the article says: Use the right tool for the job. And stop calling the tools "SQL" and "NoSQL", those terms are useless.
- danbruc 13y ago
  I completely agree, the right tool for the task is absolutely important. I would never say NoSQL stuff is bad per se [1], but I criticize its use where it is obviously a bad choice and only used because it is the new shiny thing. [1] Actually I said that in my first comment, but the wording is just bad and probably way to sensationalistic. What I really wanted to say is that it is a really bad fit for modeling you usual business with user, orders, products and the like.
frik 13y ago
In HTML5 we are stuck with NoSQL (IndexedDB), Mozilla refused to implement WebSQL. Safari, Chrome and Opera implement WebSQL using the public domain SQLite library. This means WebSQL is available on almost all smartphones/tablets and on the majority of PCs. Btw. Mozilla implemented their "NoSQL" IndexedDB on top of SQLite in the first place. Various features like bookmarks are also stored in a SQLite database. As well as in FirefoxOS. https://news.ycombinator.com/item?id=7574533 https://news.ycombinator.com/item?id=7574533
- acjohnson55 13y ago
  Serious question -- why do you need SQL in the web browser? It seems like for the amount of data a person is likely to store locally, it's not worth it. But I'm open to being proven otherwise.
  - SchizoDuckie 13y ago
    http://schizoduckie.github.io/DuckieTV/ http://schizoduckie.github.io/DuckieTV/ My user's Databases regularly exceed 3 MB. It's relational data : Serie -> Seasons -> Episodes (sometimes hundreds, sometimes thousands) Here's a 5MB example: https://dl.dropboxusercontent.com/u/44645464/DuckieTV.sqlite https://dl.dropboxusercontent.com/u/44645464/DuckieTV.sqlite The project is still in beta, so the database will most likely grow with more tables, easily linking into the existing episodes table with an actors table, for instance.
  - frik 13y ago
    Short: WebSQL enables more powerful web applications. It allows some online features to be used offline. It's not about SQL it's about a proper relational database with tables, indexes, ACID, etc. You may remember Google Gears (http://en.wikipedia.org/wiki/Google_Gears http://en.wikipedia.org/wiki/Google_Gears ), even back in 2007 one could use the Google Mail (GMail) web interface during a flight, train ride, etc. to read mails, write mails, edit drafts, etc. As soon as the connection had reestablished the data synced between client and server. Later Google rewrote it to HTML5. There was a trend to move everything to the "cloud". But with the arise of peer-to-peer web applications (WebRTC, etc) and thanks to the bad NSA news we need good client side cache and data storage APIs. I haven't seen a SQL library as shim on top of IndexedDB. It would be pervert anyway as IndexedDB is implemented on top of SQLite in Firefox. I would like to ask Mozilla to finally implement WebSQL. With a relational database, more powerful web applications with optional offline support would be possible.
climatewarrior2 13y ago
I think there is another very good reason to not use a relational database. You want to build a prototype really, really fast and your data model and organization is gonna be changing constantly. In my experience I have been able to prototype applications much faster using NoSQL solutions. I think a good approach is to "sketch" your app with NoSQL and once things start to stabilize move to an RDBMS.
- SchizoDuckie 13y ago
  This is what tooling is for: Define your entities, and generate a wrapper around a datastructure is for, just like you would do with Doctrine and PHP. Change your entity? Regenerate the wrapper and generate a migration. Problem still is: Nobody is writing tooling for WebSQL. Everybody just seems to automatically grab NoSQL, throwing away 30 years of built up software developing knowledge, including the golden rules of Database Normalisation: Don't duplicate your data, and compartmentalize.

by danbruc 13y ago

dhimes 13y ago

I'm with you here. I've found that even phones have enough juice to handle most manipulation, and the website appears more responsive because the server isn't bogged down. Haven't scaled it yet, though...

delluminatus 13y ago

I wouldn't say the model is fundamentally flawed. Looking at the roots of NoSQL (which I believe was first popularized by Google's BigTable), the original intent of a NoSQL database is to allow redundant storage of a lot of data on a DFS. They're meant for situations where you have data that doesn't fit well into a relational database model, and you need to run BIG queries (think PageRank) over the entire data in a fault-tolerant way. For instance thinking about PageRank, how many discrete data elements are in a single web page? This is "higher-order" data that in a relational database might have hundreds or thousands or tens of thousands of columns per row. Of course there are also in-memory transactional databases called NoSQL databases which are meant to be replacements for the traditional relational data store, but those are a very different beast from the original NoSQL data model.

danbruc 13y ago
That was bad and sensationalistic wording from my side. I should have said more clearly that I meant this in the context of modeling common business domains, not for specialized tasks that don't fit a relation model efficiently.

jchrisa 13y ago

One thing that is not easy to do with a relational model, is offline synchronization for high-availability on mobile devices. The document model was built for this, I find it hilarious that so few of the NoSQL databases are taking advantage of the latent capability. Try out Couchbase Mobile http://mobile.couchbase.com http://mobile.couchbase.com and compare it to best of breed relational sync.

SchizoDuckie 13y ago
It is not hard at all. people just hadn't built as much of the tools for it. Which also confuses me because we now have package managers that run on package managers. My own little WebSQL Object/Relation mapping library uses promises to get away from callback hell, and it's a pure joy to work with. It handles creating tables and inserting fixtures before the first connect promise is returned, so that you don't have to worry about the setup process and your data being available. Creating and inserting entities is as easy as > var P = new Presentation(); > P.set('name', 'test'); > P.Persist().then(function(result) { console.log('Done!') }); http://schizoduckie.github.io/CreateReadUpdateDelete.js/demo/ http://schizoduckie.github.io/CreateReadUpdateDelete.js/demo... I'm using it extensively in my Chrome plugin, and it runs on 10.000+ clients right now. Real world offline synchronisation example right here: https://github.com/SchizoDuckie/DuckieTV/blob/angular/js/services/MigrationService.js#L52 https://github.com/SchizoDuckie/DuckieTV/blob/angular/js/ser...
- frik 13y ago
  Nice work. This would be the way to go. Sadly one cannot use it everywhere as WebSQL is not implemented in Firefox nor IE :( The good thing is WebSQL works also great on mobile devices.
mamcx 13y ago
I have think about use it, but still it fail if the data is relational. I have a invoice app, and none of the NOSQL I have evaluated guarantee the order of the updates or have atomic updates (need both). Is not fun if a invoice update partially or the related tables get out of sync. In the other hand, I'm stuck in how sync with a relational model :( So, it sync well and broke my relations and transactions or it sync hard but keep my data correct.

jt2190 13y ago

> The NoSQL data model - please read as document oriented > is fundamentally flawed because it forces you into > denormalization... Can you provide an example? How does it "force" this?

danbruc 13y ago
If you want to use a purely document oriented model and not have redundancies then you have to partition your object graph into a set of documents that do not overlap and have no relationships between them. I can not think of any real example that fits this model. Start anywhere, for example, pick your users and create a document for each. Now you take the orders of each user and also put them into the document belonging to the respective user. Fine, that works. Now we come to the products they ordered, and now we have a problem. Unless no product has ever been ordered by two different users we have to duplicate product information and put it into several different documents. And there is the same problem within documents if the user repeatedly ordered the same product. All interesting problem domains just don't fit the idea of piece of JSON without references, internally or externally. There is of course support for reference in document oriented databases, but then lets just call your documents rows in a table and your references foreign keys and we are almost back at the relational model besides that documents still have a richer model, a bit like a NF² relational model or database with deep support for XML or JSON columns.
- jt2190 13y ago
  > Now you take the orders of each user and also put them > into the document belonging to the respective user. So the database doesn't force anything. The data modeler makes a questionable decision. > All interesting problem domains just don't fit the idea > of piece of JSON without references, internally or > externally. These databases won't prevent data modeling using references, they just often don't provide as much support as a relational database does for doing joins or enforcing key constraints.
  - danbruc 13y ago
    But that's my point - either you use documents in the way they are advertised and get redundancy or you normalize into many documents with relationships between them and then your documents just degenerate into entities. I am not saying there are no usecases where a document oriented model fits well, but I argue that it is not a good fit for modeling your average business.

tetha 13y ago

> The NoSQL data model - please read as document oriented These few words are pointing to a fundamental problem I'm already getting annoyed about, as little as I know about "NoSQL" databases so far: The term NoSQL-database is flawed, just as Fowler says in his introduction talk. Theres's document-oriented databases, and array-ish data storages (column family-oriented), there's graph based databases - and they have very, very different strengths. The article falls flat on that problem as well. Some data is not relational and you are correct that there is a strong overlap between document based storage systems and relational storage systems. However, I'm dealing with time series data, or with lock-graphs. I could shoe-horn those problems into a relational database, but after tinkering around with better storage engines for my problem... I'd call myself stupid for doing so. So, as the article says: Use the right tool for the job. And stop calling the tools "SQL" and "NoSQL", those terms are useless.

danbruc 13y ago
I completely agree, the right tool for the task is absolutely important. I would never say NoSQL stuff is bad per se [1], but I criticize its use where it is obviously a bad choice and only used because it is the new shiny thing. [1] Actually I said that in my first comment, but the wording is just bad and probably way to sensationalistic. What I really wanted to say is that it is a really bad fit for modeling you usual business with user, orders, products and the like.

frik 13y ago

In HTML5 we are stuck with NoSQL (IndexedDB), Mozilla refused to implement WebSQL. Safari, Chrome and Opera implement WebSQL using the public domain SQLite library. This means WebSQL is available on almost all smartphones/tablets and on the majority of PCs. Btw. Mozilla implemented their "NoSQL" IndexedDB on top of SQLite in the first place. Various features like bookmarks are also stored in a SQLite database. As well as in FirefoxOS. https://news.ycombinator.com/item?id=7574533 https://news.ycombinator.com/item?id=7574533

acjohnson55 13y ago
Serious question -- why do you need SQL in the web browser? It seems like for the amount of data a person is likely to store locally, it's not worth it. But I'm open to being proven otherwise.
- SchizoDuckie 13y ago
  http://schizoduckie.github.io/DuckieTV/ http://schizoduckie.github.io/DuckieTV/ My user's Databases regularly exceed 3 MB. It's relational data : Serie -> Seasons -> Episodes (sometimes hundreds, sometimes thousands) Here's a 5MB example: https://dl.dropboxusercontent.com/u/44645464/DuckieTV.sqlite https://dl.dropboxusercontent.com/u/44645464/DuckieTV.sqlite The project is still in beta, so the database will most likely grow with more tables, easily linking into the existing episodes table with an actors table, for instance.
- frik 13y ago
  Short: WebSQL enables more powerful web applications. It allows some online features to be used offline. It's not about SQL it's about a proper relational database with tables, indexes, ACID, etc. You may remember Google Gears (http://en.wikipedia.org/wiki/Google_Gears http://en.wikipedia.org/wiki/Google_Gears ), even back in 2007 one could use the Google Mail (GMail) web interface during a flight, train ride, etc. to read mails, write mails, edit drafts, etc. As soon as the connection had reestablished the data synced between client and server. Later Google rewrote it to HTML5. There was a trend to move everything to the "cloud". But with the arise of peer-to-peer web applications (WebRTC, etc) and thanks to the bad NSA news we need good client side cache and data storage APIs. I haven't seen a SQL library as shim on top of IndexedDB. It would be pervert anyway as IndexedDB is implemented on top of SQLite in Firefox. I would like to ask Mozilla to finally implement WebSQL. With a relational database, more powerful web applications with optional offline support would be possible.

climatewarrior2 13y ago

I think there is another very good reason to not use a relational database. You want to build a prototype really, really fast and your data model and organization is gonna be changing constantly. In my experience I have been able to prototype applications much faster using NoSQL solutions. I think a good approach is to "sketch" your app with NoSQL and once things start to stabilize move to an RDBMS.

SchizoDuckie 13y ago
This is what tooling is for: Define your entities, and generate a wrapper around a datastructure is for, just like you would do with Doctrine and PHP. Change your entity? Regenerate the wrapper and generate a migration. Problem still is: Nobody is writing tooling for WebSQL. Everybody just seems to automatically grab NoSQL, throwing away 30 years of built up software developing knowledge, including the golden rules of Database Normalisation: Don't duplicate your data, and compartmentalize.